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We, Alexandre Evfimievski, Ramakrishnan Srikant, and Rakesh Agrawal, the Applicants 
and joint inventors of the above-referenced invention defined by claims 1 -24 and disclosed in 
U.S. Patent Application Serial No. 10/624,069 hereby declare the following: 

[0001] The purpose of this declaration is to prove that we conceived the claimed 
invention prior to the August 2002 date of Exhibit A. Exhibit A is a copy of the following 
published article cited in the March 5, 2008 rejection of claims 1-24 of the present patent 
application (herein after referred to as Patent Application) under 35 U.S.C. §1 02(a): Rizvi, et al., 
"Maintaining Data Privacy in Association Rule Mining," Proceedings of the 28 th VLDB 
Conference, Hong Kong, China, 12 pages, dated August 2002 (hereinafter referred to as Rizvi). 

[0002] The following shows that we conceived our invention prior to the August 2002 
date of Rizvi, that we were diligent from the date of conception to the date of reduction to 
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practice and that we were further diligent to the date of the filing of the patent application (herein 
after referred to as Patent Application), which has a filing date of July 21, 2003. 

[0003] Proof of the conception of the claimed invention prior to August 2002 and 
diligence in reducing the invention to practice and filing the Patent Application is demonstrated 
by the attached Exhibit B in conjunction with Exhibit A. 

[0004] Exhibit B is a copy of the following published paper: Evfimievski, R. Srikant, R. 
Agrawal and J. Gehrke, "Privacy Preserving Mining of Association Rules," Proc. Of 8 th ACM 
SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (KDD), July 2002, referred to 
herein as "Privacy Preserving Mining of Association Rules" (July 2002). 

[0005] Each of the Applicants of the Patent Application are co-authors on the paper 
"Privacy Preserving Mining of Association Rules" (July 2002) along with J. Gehrke. 

[0006] J. Gehrke was a professor and advisor of A. Evfimievski, during the time period 
in which the idea for the invention was conceived. Although J. Gehrke is listed as a co-author of 
"Privacy Preserving Mining of Association Rules" (July 2002), he was not an inventor of the 
invention defined by claims 1-24 of the Patent Application. 

[0007] J. Gehrke has read the Patent Application and has declared that he is not an 
inventor of the invention defined by claims 1-24 (see Exhibit C). We, the Applicants, also 
acknowledge that J. Gehrke was not an inventor of the invention defined by claims 1-24 of the 
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Patent Application. Therefore, the portions of "Privacy Preserving Mining of Association Rules" 
(July 2002), which describe the features of claims 1-24 of the Patent Application, describe the 
Applicants' own work and no one else's. 

[0008] "Privacy Preserving Mining of Association Rules" (July 2002) describes the 
invention defined by claims 1-24. In fact "Privacy Preserving Mining of Association Rules" 
(July 2002) served as the basis for the specification, drawings and claims of the Patent 
Application. 

[0009] The following is a listing of independent claims 1, 7, 13, and 19 of the Patent 
Application that define the present invention with reference to the exemplary locations within 
"Privacy Preserving Mining of Association Rules" (July 2002), wherein the claimed feature is 
described: 

Claim 1 : A computer-implemented method of mining association rules over 
transactions from datasets while maintaining privacy of individual transactions within said 
datasets through randomization, said method comprising: 

randomizing an original dataset to create a randomized dataset [see section 4.1], said 
randomizing comprising: 

randomly dropping true items from each transaction in said original dataset[see 
section 4.1]; and 

randomly inserting false items into each transaction in said original dataset[see 

section 4.1); 
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collecting said randomized dataset in a database [see section 4.1]; 

determining support of an association rule in said randomized dataset [see section 4.2]; 

estimating support of said association rule in said original dataset based on said support 
of said association rule in said randomized dataset [see section 4.3]; and 

outputting said association rule if said support of said association rule in said original 
data set is estimated to be greater than a predetermined minimum [see section 4.5], 

wherein, due to said randomizing, privacy breaches of said individual transactions are 
controlled [see section 4.4]. 

Claim 7 : A computer-implemented method of mining association rules from 
databases while maintaining privacy of individual transactions within said databases through 
randomization, said method comprising: 

randomizing an original dataset to create a randomized dataset [see section 4.1], said 
randomizing comprising: 

randomly dropping true items from each transaction in said original dataset [see 

section 4.1]; 

randomly inserting false items into each transaction in said original dataset [see 

section 4.1]; 

collecting said randomized dataset in a database [see section 4.1]; 
mining said database to recover an association rule in said original dataset after said 
dropping and inserting processes, wherein said mining comprising [see sections 4.2-4.5]: 

determining support for said association rule in said randomized dataset [see 

section 4.2]; 
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estimating support of said association rule in said original dataset based on said 
support of said association rule in said randomized dataset [see section 4.3] ; and 

outputting said association rule if said support of said association rule in said 
original data set is estimated to be greater than a predetermined minimum [see section 4.5], 

wherein, due to said randomizing, privacy breaches of said individual transactions are 
controlled during said mining [see section 4.4]. 

Claim 13 : A computer-implemented method of mining association rules from 
datasets while maintaining privacy of individual transactions within said datasets through 
randomization, said method comprising: 

creating randomized transactions from an original dataset by [see section 4.1]: 

randomly dropping true items from each transaction in said original dataset [see 
section 4.1], and 

randomly inserting false items into each transaction in said original dataset [see 

section 4.1]; 

creating a randomized dataset by collecting said randomized transactions [see section 

4.1]; 

collecting said randomized dataset in a database [see section 4.1]; and 

mining said database to recover an association rule in said original dataset after said 

dropping and inserting processes [see sections 4.2-4.5], wherein said mining comprises: 

determining support for said association rule ms-aid- in said r andomized dataset 

[see section 4.2 J; 
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estimating support of said association rule in said original dataset based on said 
support for said association rule in said randomized dataset [see section 4.3]; and 

outputting said association rule if said support of said association rule in said 
original data set is estimated to be greater than a predetermined minimum [see section 4.5], 

wherein, due to said creating of said randomized transactions, privacy breaches of said 
individual transactions are controlled during said mining [see section 4.4]. 

Claim 19 : A computer program product on a computer-readable medium and 
tangibly embodying a program of instructions executable by a computer to perform a method of 
mining association rules from databases while maintaining privacy of individual transactions 
within said databases through randomization, said method comprising: 

randomizing an original dataset to create a randomized dataset [see section 4.1], said 
randomizing comprising: 

randomly dropping true items from each transaction in said original dataset [see 

section 4.1]; 

randomly inserting false items into each transaction in said original dataset [see 

section 4.1]; 

collecting said randomized dataset in a database [see section 4.1]; and 
mining said database to recover an association rule in said original dataset after said 
dropping and inserting processes [see sections 4.2-4.5], wherein said mining comprises: 

determining support for said association rule in said randomized dataset [see 

section 4.2]; 
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estimating support of said association rule in said original dataset based on said 
support of said association rule in said randomized dataset [see section 4.3]; and 

outputting said association rule if said support of said association rule in said 
original data set is estimated to be greater than a predetermined minimum [see section 4.5], 

wherein, due to said randomizing, privacy breaches of said individual transactions are 
controlled during said mining [see section 4.4]. 

[0010] Furthermore, dependent claims 2-6, 8-12, 14-18 and 20-24 are either explicitly 
described in "Privacy Preserving Mining of Association Rules" (July 2002) or inferred from 
details contained therein. 

[0011] "Privacy Preserving Mining of Association Rules" (July 2002) clearly predates 
the August 2002 date of Rizvi. Additionally, at the August 2002 date of Rizvi, the authors of 
Rizvi had knowledge of the details of the present invention and wrote their paper in light of that 
knowledge. This is evidenced by the fact that, as mentioned above, the details of the invention 
as defined by claims 1-24 of the Patent Application are described in "Privacy Preserving Mining 
of Association Rules" (July 2002) and further by the fact that Rizvi cites "Privacy Preserving 
Mining of Association Rules" (July 2002), as a reference, at various places throughout the text of 
the article. 

[0012] We were diligent from the date of conception in reducing the invention to practice 
and in pursuing, preparing, and filing the Patent Application. 
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[0013] On May 15, 2003, a patent attorney was instructed to prepare a patent application 
that eventually became the Patent Application. The Patent Application was eventually prepared 
and filed on July 21, 2003. 

[0014] Finally, the above declarations are made according to the best of my/our 
recollection upon review of the appropriate documents and notes, and I/we hereby acknowledge 
that willful false statements and the like are punishable by fine or imprisonment, or both (1 8 
U.S.C. 1001) and may jeopardize the validity of the Patent Application or any patent issuing 
thereon. All statements that are made herein of my/our own knowledge are true and all 
statements that are made herein based on information and belief are believed to be true. 
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Abstract 

Data mining services require accurate input, 
data for their results to be meaningful, but 
privacy concerns may influence users to pro- 
vide spurious information. We investigate 
here, with respect to mining association rules, 
whether users can be encouraged to provide 
correct information by ensuring that the min- 
ing process cannot, with any reasonable de- 
gree of certainty violate their privacy. We 
present a scheme, based on probabilistic dis- 
tortion of user data, that can simultaneously 
provide a high degree of privacy to the user 
and retain a high level of accuracy in the min- 
ing results. The performance of the scheme is 
validated against representative real and syn- 
thetic datasets. 

1 Introduction 

The knowledge models produced through data mining 
techniques are only as good as the accuracy of their in- 
put data. One source of data inaccuracy is when users 
deliberately provide wrong information. This is espe- 
cially common with regard to customers who are asked 
to provide personal information on Web forms to e- 
commerce service providers. The compulsion for doing 
so may be the (perhaps well-founded) worry that the 
requested information may be misused by the service 
provider to harass the customer. As a case in point, 
consider a pharmaceutical company that asks clients 
to disclose the diseases they have suffered from in or- 
der to investigate the correlations in their occurrences 
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for example. "Adult Females with malarial infections 
are also prone to contract tuberculosis". While the 
company may be acquiring the data solely for genuine 
data mining purposes that would eventually reflect, it- 
self in better service to the client, at the same time 
the client might worry that if her medical records are 
either inadvertently or deliberately disclosed, it may 
adversely affect her employment opportunities. 

We investigate, in this paper, whether customers 
can be encouraged to provide correct information by 
ensuring that the mining process cannot, with any rea- 
sonable degree of certainty, violate their privacy. At 
the same time, we would like the mining process to 
be as accurate as possible in terms of its results. The 
difficulty lies in the fact that these two metrics: pri- 
vacy and ace uracil, are typically contradictory in na- 
ture, with the consequence that improving one usu- 
ally incurs a cost in the other [1]. Therefore, we com- 
prise on the ideal and perhaps infeasible goal of having 
both complete privacy and complete accuracy through 
approximate solutions that provide practically accept- 
able values for these metrics. Note further that since 
the purpose of data mining is essentially to identify 
statistical trends, cent-per-cent accuracy in the mining 
results is perhaps often not even a required feature. 

Our study is carried out in the context of extract- 
ing association rules from large historical databases, 
a popular mining process [2] that identifies interest- 
ing correlations between database attributes, such as 
the one described in the pharmaceutical example. For 
this framework, we present a scheme called MASK 
(Mining Associations with Secrecy Konstraints) , that 
attempts to simultaneously provide a high degree of 
privacy to the user and retain a high degree of accu- 
racy in the mining results. Our scheme is based on a 
simple probabilist ic distortion of user data, employing 
random numbers generated from a pre-defined distri- 
bution function. It is this distorted information that 
is eventually supplied to the data miner, along with a 
description of the distortion procedure. We define a 
privacy metric and present an analytical formula for 
evaluating the privacy obtained under I his metric by 
the distortion approach. 



A special feature of our scheme is that the distortion 
process can be i si] mplenif ted at the data source 
itself, that. is. at the user machine. This increases 
the confidence of the user in providing accurate infor- 
mation since she does not have to trust a third-party 
to carry out the distortion process before the data is 
acquired by the service provider. Note that some of 
the other privacy techniques suggested in the litera- 
ture, such as swapping values between records [7], do 
not support this feature since they require the entire 
database to be available for their functioning. 

As described in detail later in the paper, min- 
ing the distorted database can be, apart from being 
error-prone. signilieantly more expensive in terms of 
both time and space as compared to mining the true 
database. We present a variety of optimizations to 
address these issues. 

Finally, the performance of MASK's mining scheme 
is validated against representative real and synthetic 
datasets, with respect to both privacy and accuracy. 
Our results indicate that there are regions of the dis- 
tortion parameter space that are conducive to satis- 
factorily meeting the dual objectives. 

1.1 Organization 

The remainder of this paper is organized as follows: 
In Section 2, we describe the privacy framework em- 
ployed in our study and in Section 3, we quantify the 
privacy attained under this framework by our distor- 
tion method. Then, in Section 4, we present our new 
MASK algorithm for mining the distorted database. 
Optimizations to improve the space and time com- 
plexity of MASK are described in Section 5. The 
performance model and the experimental results are 
highlighted in Sections 6 and 7, respectively. Bounds 

ing process are given in Section 8. Related work on 
privacy-preserving mining is reviewed in Section 9. Fi- 
nally, in Section 10, we summarize the conclusions of 
our study and outline future avenues to explore. 

2 Problem Framework 

In this section, we describe the framework of the pri- 
vacy mining problem that we consider here. 

2.1 Database Model 

We assume that each customer contributes a tuple to 
the database with t he tuple being a fixed-length se- 
quence of l's and 0's. A typical example of such a 
database is the so-called "market-basket" database [2] 
wherein the columns represent the items sold by a su- 
permarket, and each row describes, t hrough a sequence 
of l's and 0's, the purchases made by a particular cus- 
tomer (1 indicates a purchase and 0 indicates no pur- 
chase). We also assume that the overall number of l's 



in the database is signifii mth sm Lie: than the num- 
ber of 0's — this is especially true for market-baskets 
since each customer typically buys only a small frac- 
tion of ah the items available in the store. In short, 
the database is modeled as a large disk-resident two- 
dimensional sparse boolean matrix. 

Xote that the boolean representation is only logical 
and that the database tuples may actually be physi- 
cally stored as "item-lists" , that is, as an ordered list 
of the identifiers of the items purchased in the trans- 
action. The list representation ma}' appear preferable 
for the sparse databases that we are considering, since 
it reduces the space requirement as compared to stor- 
ing entire bit-vectors. However, because of the fact 
that we are distorting user information, it may be the 
case that the distorted matrix will not be as sparse 
as the true database. Therefore, in this paper, we as- 
sume that the distorted database in Mored as a, large 
collection of bit- vectors. 

2.2 Mining Objectives 

The goal of the miner is to compute association rules 
on the above database. Denoting the set of transac- 
tions in the database by T and the set, of items in the 
database by 1, an association rule is a (statistical) im- 
plication of the form X y, where X, y C I and 
X C\Y = <j>. A rule X =$> y is said to have a support 
(or frequency) factor s iff at least s% of the transac- 
tions in T satisfy Xuy. A rule X y is satisfied in 
the set of transactions T with a confidence factor c iff 
at least c% of the transactions in T that satisfy X also 
satisfy y. Both support and confidence are fractions 
in the interval [0,1]. The support is a measure of sta- 
tistical significance, whereas confidence is a measure 
of the strength of the rule. 

A rule is said to be "interesting" if its support 
and confidence are greater than user-defined thresh- 
olds sup min and con m j n , respectively, and the objec- 
tive of the mining process is to find all such interesting 
rules. It has been shown in [2] that achieving this goal 
is effectively equivalent to generating all subsets X of X 
that have support greater than sup min - these subsets 
are called frequent itemsets. Therefore, the mining ob- 
jective is, in essence, to efficiently discover all frequent 
itemsets that are present in t he dat abase. 

2.3 Privacy Metric 

As mentioned earlier, the mechanism adopted in this 
paper for achieving privacy is to distort the user data 
before it is subject t o t he mining process. Accordingly, 
we measure privacy with regard to the probability with 
which the user's distorted entries can be reconstructed. 
While privacy could be measured at t lie granularity of 
entire tuples, we consider here the stronger require- 
ment of ensuring privacy at the level of individual en- 
tries in each customer tuple. In short, our privacy 



metric is: "With what probabilh < 1 a given 1 or 0 
in the true matrix be reconstructed"? 

A related issue here is whether the user would 
want the same level of privacy for both l's and O's? 
For many applications, such as the market-basket 
database, it appears reasonable to expect that cus- 
tomers would want more privacy for their l's than for 
their O's, since the l's denote specific actions whereas 
the O's are the default options. 

3 Quantifying MASK's Privacy 

In this section, we present the distortion procedure 
used by the MASK scheme and quantify the privacy 
prov ided by the procedure, as per the above metric. 

3.1 Distortion Procedure 

A customer tuple can be considered to be a random 
vector X = {Xi}, such that Xi = 0 or 1. We generate 
the distorted vector from this customer tuple by com- 
puting Y = distort (X) where Yj = Xi XOR r t and 
Ti is the complement of r L . a random variable with den- 
sity function f(r) = bernoulli(p) (0 < p < 1). That 
is, rj takes a value 1 with probability p and 0 with 
probability 1 — p. 

The net effect of the above computation is that the 
identity of the i th element in X is kept the same with 
probability p arid is Jlipped. with probability ( I — //). 
All the customer tuples are distorted in this fashion 
and make up the database supplied to the miner - in 
effect, the miner receives a probabilistic function of the 
true customer database. 

Note that, in principle, it is possible to use different 
settings of p for distorting different items. That is, to 
have a vector of p settings ranging across the columns 
of the database. For simplicity, we will assume here 
that a single p is used for all the items - this choice also 
has useful implementation implications, as described 
later in Section 5. 

We now move on to quantifying the privacy ob- 
tained by the above distortion procedure. In the fol- 
lowing analysis we first consider the extreme case, 
where the user wants to have maximum privacy for 
l's but is completely unconcerned about the O's. Af- 
ter that, we derive the general privacy equations where 
the customer, though more conservative about l's, re- 
quires a degree of privacy for the O's too. 

A caveat: Our privacy estimates do not take into ac- 
count the fact t hat t here may be a reduct ion in privacy, 
as pointed out recently in [1, 8], when the mining out- 
put (i.e., the associat ion rules) is used to re-interrogate 
the distorted database - we plan to investigate this is- 
sue in our future work. 

3.2 Reconstruction Probability of a 1 

Let Si be the true support of item i, normalized to the 
number of tuples in the database. This means that the 



probability that a random customer C bought this i th 
item is Sj. We now have to evaluate the probability 
that given that C indeed did buy item i, her original 
T can be reconstructed from the distorted entry. De- 
noting the original entry as Xi and the distorted entry 
as Yi. the probability of correct reconstruction is given 
by: 

fti(p,*<) = 

P r {Yi = l\Xi = l}x P r {Xi = l\Yi = 1} 
+ 

P r {Yi = 0\Xi = 1} x P r {Xi = l\Yi = 0} 

This expression captures the "round-trip" of go- 
ing from the true database to the distorted database 
and then returning to guess the contents of the true 
database. It can be simplified to 

n 1 (p,s i ) = 

pxP r {Xi = l\Yi = l} 
+ 

(1 -p) x P r {Xi = l\Yi = 0} 
But, we know that 

Pr{X l = \\Y l = \}= PAX ^Jf 1} 

- ^{^,=i}xp,{y i =i|x i =i> 

Pr{Y, = l} 

SjXp 

/', |.v i ! /'. \ ) i|.v i ! /'. j.v o}x/', {>-, i|.v, o] 

= SiXp + (l-Si)x(l-p) 

Similarly, 

P r {x i = m = o} = StX{1 ^_ Si)xp 

Putting it all together, we obtain 

*i(p, ■*) = SlXP+( ^; x(1 - P) + SiX( r P) vi)x P 

The above expression reflects the reconstruction 
probability of a T' in a random item i. To find a total 
measure of reconstruction, we range across all items: 

Ki(p)= S ' S ^' Si) (1) 
The above expression is minimized when all the 
items in the database have the same support, and in- 
creases wit h t he variance in the supports across items. 
As discussed later in Section 7, with the appropriate 
choice of p, this increase is marginal for the market- 
basket type of datasets that we consider here. There- 
fore, as a first-level approximation, we replace the 
item-specific supports in I lie above equation by s'o, the 
average support of an item in the database. With this, 
the reconstruction probability simplifies to 

U ^P) = ,„ l" u I V ' ~ i n p (2 ) 



sO=0.01 /. 




Figure 1: Reconstruction Probability 7£i(p) 

We can plot 7^i(p) as a function of p for different 
values of s 0 , as shown in Figure 1. We observe here 
that: 

1. The riH-oiis1.ruc1.ion probability is high al I he ex- 
tremes and lowest at the center (i.e p = 0.5). 
This is 1,o l)o intuitively expected since setting 
p = 0.5 imparts the maximum randomness to the 
distorted values. 

2. The curves are symmetric around p = 0.5. The 
implication of this is that there is no difference, 
reconstruction-wise, between choosing a value p 
or its counterpart 1 — p. This may appear sur- 
prising at first glance since a matrix that is dis- 
torted with p = 0.1 would tend to "look" very 
different with regard to the true matrix, as com- 
pared to the same matrix distorted with p = 0.9. 
However, recall that the data miner is also pro- 
vided with a description of the distortion proce- 
dure, that is, he knows the value of p. In this 
situation, mere differences in appearance do not 
result in any additional privacy. A practical use 
of this feature, however, is in psychological terms: 
Distorting with a low value of p as opposed to 
its high- valued complement, might be more com- 
forting to the user since at least visually it will 
appear to be considerably different from the true 
information that she had supplied. 

3. Although the minimum always is at p = 0.5, the 
curves become natter as the average support of 
items decreases, for a typical market-basket type 
dal abase wil h an average t ransaet ion length of 10 
and the number of items being 1000, the average 
support is 0.01, which corresponds to the lowest 
curve in Figure 1. 

In the above derivation, it may appear unintuitive that 
the reconstruction probability depends on the support 
of items. The reason for this is the following: We are 
considering the possibility of reconstruction of the true 
value of an entry given the distorted entry. If the data 



miner gets a '1' (or a '0') for a particular entry in the 
distorted database, t he probability 1 hat it came from a 
'1' in the true database not only depends on p but also 
on the distribution of Fs and O's in the true database. 

Yet another issue is that we have used the true sup- 
ports of items in the derivation, but these values are 
not known to the data miner. Therefore, it may appear 
that we are overestimating the reconstruction proba- 
bility. However, the point is that since the ultimate 
goal is to be able to mine the distorted database cor- 
rectly, we make the conservative assumption that the 
miner will be able to derive reasonably accurate item 
supports, implying that he dot ,s have access to the .s ; 
values. 

3.3 The General Reconstruction Equation 

We now move on to deriving the relationship between 
p and the reconstruction probability for the general 
case where the customer may wish to protect both hel- 
l's and O's, but her concern to keep the Fs private is 
more than that for the O's. 

Analogous to the manner in which we computed 
Hi(p) above, we can derive the probability with which 
a '0' can be reconstructed as: 

n 0 [p, Si ) = 

P r {Yi = l\Xt = 0} x P r {Xi = 0\Yi = 1} 
+ 

P r {Yi = 0\Xi = 0} x P r {Xi = 0\Yi = 0} 
leading to 

K 0 (P) - (1 _ So)xp+SoX(1 _ p) + SoXp+( i_ So)x( i_ p) 

Our aim is to minimize a weighted average of TZi(p) 
and TZo (p). This corresponds to minimizing the proba- 
bility of reconstruction of both l's and O's. The weight 
denotes the preference which the privacy of 1 's has over 
that of O's. The total reconstruction probability, TZ(p), 
is then given as 

1Z(p) = oWi(p) + (1 - a)ft 0 (p) (3) 

where a is the weight given to l's over O's. Note that 
the a setting must incorporate the fact that the num- 
ber of O's in t he dal abase is more I ha.n t hat of I 's. So, 
for example, if we set a = 0.5 for a database that has 
s 0 = 0.01, we are indicating that the privacy of l's is 
99 times more critical than that of O's (as the number 
of O's is 99 times more than that of l's). 

3.4 Privacy Measure 

Armed with the ability to compute the reconstruction 
probability, we now simply define user privacy as the 
following percentage: 

rip) = (i - n{ P )) * ioo. (4) 
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Table 1: Privacy attained with s 0 = 0.01 and a = 0.9 

That is, when the reconstruction probability is 0. the 
privacy is 100%, whereas it is 0 if the TZ(p)= 1. In 
Figure 2. we plot, this user privacy as a function of p 
for .so 0.01 with different, values of a. Note that for 
a given value of s 0 , the shape of the curve is fixed, and 
it is only the value of a that decides the absolute value 
of the attained privacy. 
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Figure 2: Privacy V(p) attained for so = 0.01 

Further, note that the curves have the "knee- 
points" at p = 0.1 and p = 0.9 and that for a = 0.9, the 
privacy is almost constant at a high value of around 
85% in this large range (0.1 to 0.9) of distortion proba- 
bilities - the explicit values are shown in Table 1 . This 
result is very encouraging since it means that we now 
have considerable flexibility in choosing the p value - 
in particular, we can choose it in a manner that will 
minimize the error in the subsequent mining process. 

4 Mining the Distorted Database 

Having established the privacy obtained from our dis- 
tortion procedure, we now move on to presenting 
MASK's technique for estimating the true (accurate) 
supports of it cutsets From a distorted database. Later, 
in Section 5, we present a variety of opt imizat ions t hat 
help to speed up the estimation process, filially, in 
Section 7. we evaluate t he quality of t hese est iniat ions. 

In the following discussion, we first show how to es- 
timate the supports of 1-itemsets (i.e. singletons) and 
then present the general n-itemset support estimation 
procedure. In this derivation, it is important to keep 
in mind that the miner is provided with both the dis- 



torted matrix as well as the distortion procedure, that 
is, he knows the value of p that was us< 1 in distorting 
the true matrix. 

4.1 Estimating Singleton Supports 

We denote the original true matrix by T and the dis- 
torted matrix, obtained with a distortion probability 
of p. as D. Now consider a random individual item 
i. Let cj and Cq represent the number of l's and 0's, 
respectively, in the i column of T, while and 
represent the number of l's and 0's. respectively, in 
the i column of D. W ith this notation, wo estimate 
the support of i in T using the following equation: 

C T = M- 1 C D (5) 

Ma V]-=[$k=[f] 

The M matrix in the above equation incorporates 
the observation that by our method of distortion, if a 
column had n l's in T, these l's will generate approx- 
imately pn l's and (1 — p)n 0's for the same column in 
D. Similarly for the 0's of this column in T. There- 
fore, given cf and Cq , it is possible to estimate the 
value of cj, that is, the true support of item i. 

Note also that the equation rules out the possibility 
of using p = 0.5 because at this value the matrix be- 
comes singular and is not inverfible. Intuitively, this 
happens because at this value of p, the matrix C D 
does not carry sufficient information to be able to re- 
construct the values of cj and Cq. 

4.2 Estimating n-itemset Supports 

It is easy to extend Equation 5, which is applicable 
to individual items, to compute the support for an 
arbitrary n-itemset. For this general case, we define 
the matrices as: 




c D c T 

„D T 

C 0 J L Cq 



Here c\ should be interpreted as the count of the 
tuples in T that have the binary form of k (in n digits) 
for the given itemset (that is, for a 2-itemset, c\ refers 
to the count of 10's in the columns of T corresponding 
to that itemset, to the count of ll's, and so on). 
Similarly, is defined for the distorted matrix D. 

Finally, the matrix M is defined as: 

rrii j = The probability that a tuple of the form 
corresponding to cj in T goes to a tuple 
of the form corresponding to cf in D 



For example, for a 2-itemset is the probability 

that a 10 tuple distorts to a 01 tuple. Accordingly, 
mi 2 = (1 — — P)- The basis for this formulation 
lies in the fact that in our distortion procedure, the 
component columns of an n-itemset are distorted in- 
dependently. Therefore, we can use the product of the 
probability terms. 

At first glance, it might seem that the above mining 
process may need an exponential number of counters 
(2™ counters for an n-itemset). making the process in- 
feasible in practice. But. in fact, this is not really 
so if the same value of p is used for all items. This 
is because it results in the matrix M having inter- 
esting symmetry properties that are reflected in M 1 
also. In particular, we show in Section ■"> how a linear 
number of counters (specifically, n + 1 counters for an 
n-itemset) are now sufficient to complete the mining 
process. 

4.3 The Full Mining Process 

The above equations help us to estimate the value 
of c|"„_ 1 for an n-itemset by using the values of cf, 
0 < i < 2 n — 1. But, first we need to compute the cf 
values themselves. For this purpose, in principle we 
could use, after the modifications described below, any 
of the numerous association rule mining algorithms 
proposed in the literature (e.g. [3, 13, 9, 16]). With a 
view to simplicity, we have currently implemented our 
system based on the classical Apriori algorithm [3]. 
Apriori is a multi-pass algorithm wherein the i th pass 
computes the frequent i-itemsets by counting all candi- 
date itemsets associated with the pass and, after each 
pass, the AprioriGen algorithm is used for generat- 
ing the candidate itemsets for the next pass. In our 
approach too, the i th pass identifies large i-itemsets, 
and the AprioriGen algorithm is used for generating 
the candidate itemsets for the next pass. 

A critical difference between our approach and that 
of Apriori, however, is the following: Consider that 
we are counting, say, 2-itemsets. Here, Apriori only 
needs to keep track, for each candidate 2-itemset, of 
the number of tuples in which there was a T' for both 
the items appearing in the itemset. That is, it needs to 
count only the 'll's. But, in our case, we need to keep 
track of all combinations: 00, 01, 10, and 11. This is a 
direct fallout of the fact t hat we have distorted t he t rue 
matrix, and an original '11' can now potentially be any 
of the four combinations. We describe in Section 5 a 
simple optimization that can significantly reduce the 
total amount of counting. 

Another important point to note here is that the 
equation to compute the true supports needs to be 
evaluated only at the end of every pass over the dis- 
torted matrix, and not on a tuple-by-tuple basis. Fur- 
ther, if the same p value is used for all columns, the 
matrix M is identical for all candidate n-itemset s and 
therefore has to be generated only once at the end of 



each pass. Finally, note that the size of the square 
matrix is 0(2") for candidate n- itemsets. 

5 MASK Mining Optimizations 

We now describe a set of optimizations to improve the 
efficiency of the mining process described in the pre- 
vious section. 

5.1 Linear Number of Counters 

We first consider how to reduce the number of counters 
required for each itemset. At the end of the n th pass, 
we generate a square matrix of size 2™ which depends 
only on p. We then invert it and mult iply the resulting 
matrix with the counts of the 2" components of every 
n-itemset. In effect, the reconstructed support is a 
weighted sum of the counts of all 2™ components in 
the distorted database. A close observat ion will reveal, 
however, that these 2" weights have only n I I distinct, 
weights in them. 

For example, for a 2-itemset. we have the estimated 
reconstructed support value to be 

Seat = aiC^ + a 2 C° + a 3 C° + a 4 C£ 

where C® y is the count of xy tuples in the distorted 
database, and the a$ are the associated weights. Here, 
the weights and az will be equal because the proba- 
bility that a '11' distorts to a TO' is equal to the prob- 
ability that a '11' distorts to a '01' (both are p(l —p)). 
Hence, the reverse component weights are also equal. 
Therefore, overall, we need to maintain only 3 coun- 
ters - one each for 00 's and ll's, and a third which is 
common for 01's and 10's. 

The above observation can be generalized to an n- 
itemset. For the 2™ counts we have merely n + 1 dis- 
tinct weights: One for '00. .0', one for '11. .1' and one 
each for components that have the same number of 
l's (or equivalently 0's) in them. For example, for a 
3-itemset, only 4 counters are required: one each for 
000 and 111, one for the triplet (001,010,100) and the 
fourth for the triplet (011,101,110). 

5.2 Reducing Amount of Counting 

Apart from reducing the number of counters, we can 
also achieve some reductions in the amount of counting 
by making use of simple algebraic properties. 

For example, consider 2-itemsets: The counting 
of only ll's takes 0(tlen 2 ) operations, where tlen is 
the transaction length. However, counting all 4 com- 
ponents (00,01,10,11), which we have to do for the 
distorted matrix, takes 0(|Candidates|) operations. 
Since the number of candidate 2-itemsets is typically 
very large, the latter will take much more time than 
the ordinary mining process. 

We can optimize the above by using the observation 
that c^+c^+c^+c^ must be equal to the database 



cardinality, dbsize. This means that we can choose 
to not count one of these components, say '00's. since 
it is derivable from the other counts. In conjunction 
with the counting process described below, this opti- 
mization can result in considerable savings. 

Our counting process examines the (distorted) cus- 
tomer tuples one by one. For the current customer 
tuple, the purchase vector of l's and O's is converted 
into an item-list that contains only the identifiers of 
the items that have a '1' in the transaction. From this 
list, the identifiers of all the (previously estimated) 
infrequent 1-itemsets are removed. The next stage 
is to create a complement-list that consists of ah the 
(previously estimated) frequent 1-itemsets that do not 
appear in this transaction. Let the item-list and the 
complement- list be of lengths mi and 7712, respectively, 
(with mi + "m-2 =| |, the total number of frequent 
1-itemsets). If we now restrict our counting to the 
ll's, Ol's and 10's, it will take 0(mf )+0(m 1 m 2 ) op- 
erations. For high settings of the \> parameter, the 
value of mi will be rather small and the transaction 
will have a large number of '00' pairs. The approach 
of not counting the DO's can therefore result in signif- 
icant savings in such situations. In fact, we observed 
in our experiments (described in Section 7) that for 
p = 0.9, the execution time for Pass 2 reduced by a 
factor of four due to this optimization. 

6 Performance Framework 

The privacy of the MASK scheme was analytically 
evaluated in Section 3. We now move on to evalu- 
ating its accuracy with regard to the mining results 
that it derives from the distorted database. 

Since MASK is a probabilistic approach, fundamen- 
tally we cannot expect the reconstructed support val- 
ues to co-incide exactly with the actual supports. This 
means that we may have errors in the estimated sup- 
ports of frequent itemsets with the reported values be- 
ing either larger or smaller than the actual supports. 

Errors in support estimation can have an even more 
pernicious effect than just wrongly reporting the sup- 
port of a frequent itemset. They can result in errors 
in the identities of the frequent itemsets. This be- 
comes especially an issue when the sup min setting is 
such that the support of a number of itemsets lie very 
close to this threshold value. Typically, this happens 
for lower values of sup min due to the larger number of 
frequent itemsets in the database. Such "border-line" 
itemsets may get wrongly reported as either frequent 
or rare, based on how I he prohahilisl ic evalual ion est i- 
mates their supports. That is, we can encounter both 
false positives and falsi negatives. 

Worse, errors in association rule mining percolate 
t li rough the various passes of I he mining process t hat 
is, an error in identifying a 1-itemset correctly has a 
ripple effect in terms of causing errors in the remainder 
of the frequent itemset lattice. 



To assess the above effects, we evaluate the min- 
ing process under two conditions: The first with the 
sup-min value provided by the user, and the second with 
a marginally lower value. The lower value is expected 
to help reduce the number of false negatives at the risk 
of increasing the number of false positives, but this is 
in keeping with the view that it is more important to 
have coverage as compared to precision. For the ex- 
periments described here, we evaluate the impact of a 
109c reduction, denoted r 109c. in the sup min value. 

To quantify the errors that we are making, we com- 
pare our outputs with those derived from Apriori run- 
ning on the true database with both the user provided 
supmin. as well as with the lowered value mentioned 
above. 

6.1 Data Sets 

Our evaluations were carried out on two representative 

databases: 

1 . A synthetic database generated from the IBM Al- 
maden generator [3]. The synthetic database was 
created with parameters T 10.1 1.1)1 M.N I K (as per 
the naming convention of [3]), resulting in a mil- 
lion customer tuples with each customer purchas- 
ing about ten items on average. 

2. A real dataset, BMS- Web View- 1 [20], placed in 
the public domain by Blue Martini Software. This 
database contains click-stream data from the web 
site of a (now defunct) legwear and legcare re- 
tailer. There are about 60,000 tuples with close to 
500 items in the schema. In order to ensure that 
our results were applicable to large disk-resident 
databases, we scaled this database by a factor of 
ten, resulting in approximately 0.6 million tuples. 

The measured values of so for these two databases 
turned out to be 0.01 and 0.005, respectively. 

6.2 Support and Distortion Settings 

We evaluated the mining accuracy of MASK on the 
above datasets for a variety of sup min and p values. 
We present here the results for two xiip miu values, 
namely 0.25% and 0.5%. The 0.25% sup min value rep- 
resents, in a sense, the "worst-case" environment for 
our algorithm due to the presence of a large number 
of border-line itemsets. 

The p values we consider are p = 0.9 and p = 0.7 
(recall that these are equivalent to p = 0.1 and p = 0.3, 
respectively, with regard to privacy). For these set- 
tings, the associated privacy values for l's, as com- 
puted from Equation 1, are 85%. and 96%, respectively, 
for the synthetic database, and 89% and 97%, respec- 
tively, for the real database. 



6.3 Error Metrics 

We evaluate two kinds of mining errors, Support Error 
and Identity Error, in our experiments: 

Support Error (p) : 

This metric reflects the (percentage) average rela- 
tive error in the reconstructed support values for 
those itemsets that are correctly identified to be 
frequent. Denoting the reconstructed support by 
recsup and the actual support by actsup, the 
support error is computed over all frequent item- 
sets as 

1 | recsup f - act-sup f \ 
| / | actsupf 

We compute this metric individually for each level 
of itemsets, that is, for 1-itemsets, 2-itemsets, etc. 

Identity Error (<r) : 

This metric reflects the percentage error in identi- 
ly'mg frequent itemsets and has two components: 
<t + , indicating the percentage of false positives, 
and a~ indicating the percentage of false nega- 
tives. Denoting the reconstructed set of frequent 
itemsets with R and the correct set of frequent 
itemsets with F, these metrics are computed as: 

<J+ = * 100 a- = * 100 

7 Experimental Results 

We now present the results of our experiments con- 
ducted under the framework described above. The 
results for the synthetic database are presented first, 
followed by those for the real database. 

7.1 Synthetic Database 

Experiment 1: p=0.9, sup min =0.25%, r=0% 

Our first experiment was conducted on the syn- 
thetic database with a distortion parameter of p=0.9, 
sup m i n =0. 25%, and no relaxation. As mentioned ear- 
lier, this experiment represents, in a sense, the worst 
case scenario for MASK due to the extremely low 
sup m i n value. 

The results for this experiment, are shown in Table 2 
- in this table, the level indicates the length of the 
frequent itemset, | F | indicates the number of frequent 
itemsets at this level, and the other three columns are 
the error metrics defined in the previous section. 
The results indicate I hat first ly t he support error (p) is 
reasonably small, less than .">'•<' at all levels. Secondly, 
the negative identity error is also small, not exceeding 
6% at the maximum. Finally, the positive identity er- 
ror is also in the same range. Note that the maximum 
errors occur for the 2-itemsets, which is as per expec- 
tations since the number of itemsets is the maximum 
at this level. 
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Table 2: p=0.9.sup mhl 0.25% .r 0%,Synthetic 

Experiment 2: p=0.9, sup min =0.25%, r=10% 

Our second experiment evaluated the effect of 
marginally relaxing the sup ln ; n value by 10% - that 
is, using a sup min of 0.225% instead of 0.25%, keeping 
the rest of the parameters the same as that of Kxper- 
iment 1. The p and a results for this experiment are 
shown in Table 3. 



Level 


\F\ 


P 






1 


6N9 


3.37 


0.7:! 


3.19 


2 


2<i|N 


3.73 


0.19 


19.68 


3 


1990 


1.76 


0 


28.09 


4 


1118 


1.29 


0 


25.81 




730 


1 .32 


0 




(i 


212 


1.37 


0 


25.47 


7 


35 


1,10 


0 


51.43 


8 


3 


0.99 


0 


66.67 



Table 3: p=0.9,sup min =0.25%,r=10%, Synthetic 

We see here that while the support error shows little 
change as compared to the previous experiment, the 
negative identity error goes down very significantly, 
becoming less than 1%, achieving the desired goal. 
The price to pay for this, however, is the substantial 
increase in the positive identity error. The marked 
change in the values of the identity errors also high- 
lights the significant presence of "border-line" itemsets 
in the database. 

Experiment 3: p=0.9, sup min =0.5%, r=0% 

We now move on to repeating Experiment 1 for an in- 
creased sup m i n value of 0.5, corresponding to a "nicer" 
environment for MASK. The results of this experiment 
are shown in Table 4. 

The results here show that the errors generally reduce 
as compared to Experiment 1 due to the sparser distri- 
bution of frequent itemsets. Further, 2-itemsets con- 
tinue to be associated with the maximum error. 

Experiment 4: p=0.9, sup min =0.5%, r=10% 

Our next experiment evaluated the effect of marginally 
relaxing the sup min value by 10% - that is, using a 
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Table 4: p=0.9,sup min =0.5%,r=0%,Synthetic 

■KiPmin of O.-I-VX instead of O.'M keeping the rest 
of the parameters the same as that of Experiment 3. 
The p and a results for this experiment are shown in 
Table 5. 
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Table 5: p=0.9,sup min =0.5%,r=10%,Synthetic 

Similar to Experiment 2, the results here too indicate 
that a marginal relaxation can almost completely elim- 
inate the false negative error component, ensuring that 
none of the true frequent itemsets are missed. At the 
same time, the false positive error goes up considerably 
since trying to "catch all the good fish" inevitably also 
attracts unwanted material. 



Experiment 5: p=0.7, sup min =0.25%, r=10% 

The previous experiments were all run at a distortion 
probability ofp = 0.9 corresponding to a privacy factor 
of 85%. We now consider the possibility of improving 
the privacy measure to 96' { by {•hanging lo /) 0.7 and 
evaluate the impact of this change on the accuracy. 
The results for this experiment are shown in Table 6. 
We see from these results that there is a dramatic in- 
crease in all the error metrics. For example, the sup- 
port error goes up to about 25'/r on average, while 
the identity errors are huge. In fact, the errors go up 
to the extent that the results produced by the mining 
process are essent ially meaningless. The implication is 
that privacy and accuracy represent an extremely sen- 
sitive tradeoff and that there is only a small parameter 
region wherein we can hope to obtain reasonable val- 
ues for both metrics. 
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Table 6: p=0.7,sup min =0.25%,r=10%, Synthetic 



7.2 Real Database 

We now move on to experiments conducted on the real 
database which, as mentioned earlier, contains infor- 
mation about the click-stream logs of an online leg- 
wear manufacturer. For ease of comparison, those ex- 
periments modeled exactly the Mime environments as 
thono evaluated for the synthetic database that is. 
Experiments 6 through 10 presented below correspond 
one-to-one with Experiments 1 through 5. 

Experiment 6: p=0.9, sup min =0.25%, r=0% 
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Table 7: p=0.9,sup min =0.25%,r=0%,Real 
The results of this experiment are shown in Table 7. 

ably low, are somewhat higher than the corresponding 
numbers for the synthetic database. But, in fact, the 
absolute number of errors is fewer and it is due to the 
much smaller number of frequent itemsets that the ef- 
fects of these errors are magnified. For example, the 
number of 2-itemsets in the real database is an order of 
magnit ude smaller t han that in the synthetic database. 

Experiment 7: p=0.9, sup min =0.25%, r=10% 
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Table 8: p=0.9,sup min =0.25%,r=10%,Real 

The results of this experiment are shown in Table 8. 
We observe that these results are similar to those of 
Experiment 2 - the basic errors are low and relaxation 



significantly reduces the negative identity error at an 
attend: il at 1 ease in the positive identity error. 

Experiment 8: p=0.9, sup min =0.5%, r=0% 



Level 


\F\ 


P 


a 




1 


150 


4.23 


0.67 


4.67 


2 


45 


2.42 


2.22 


4.44 


3 


6 


1.07 


0 


16.66 



Table 9: p=0.9,sup min =0.5%,r=0%,Real 

The results of this experiment are shown in Table 9. 
Here, we see that the stricter support threshold results 
in a very small set of frequent itemsets and that the 
errors are acceptably low. 

Experiment 9: p=0.9, sup min =0.5%, r=10% 
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Table 10: p=0.9,sup min =0.5%,r=10%,Real 

The results of this experiment are shown in Table 10. 
We see that the relaxation completely removes all false 
negatives, with the expected attendant increase in the 
false positives. 

Experiment 10: p=0.7, sup min =0.25%, r=10% 



Level 


\F\ 


P 






1 


249 


18.96 


7.23 


15.66 




239 


33.59 


20.08 


1907.53 


3 


73 


32.87 


30.14 


2308.22 


4 


4 




50 


400 



Table 11: p=0.7,sup min =0.25%,r=10%,Real 

The results of this experiment are shown in Table 11. 
These results are similar t.o those seen in Kxperiment 
5 - the reduction in distortion probability results in 
a disproportionate increase in the errors for both the 
support and the identity metrics. 

7.3 Summary 

Overall, our experiments indicate that by a careful 
choice of distortion probability, it is possible to simul- 
tancouslv achieve satisfactory privacy and accuracy. 
In particular, they show" that there is a small "window 
of opportunity" around the p = 0.9 value where these 
dual goals can be met. Moving away from this window 



towards lower values of p, however, results in skyrock- 
eting errors, while in< n sin tin i Aue of p will result 
in significant loss of privacy. 

We have conducted several other experiments with 
different market-basket type databases and the results 
are consistent with those presented here. One other 
issue is the running time of our mining algorithm. As 
mentioned earlier, mining the distorted database is 
intrinsically significantly more expensive than mining 
the real database. Currently, we have not yet imple- 
mented all the optimizations described in Section 5, 
and we have therefore refrained from presenting the 
running times since these numbers will change when 
all the optimizations are in place. The current situ- 
ation is that we are able to mine the real database 
at 0.25% support in about 30 minutes on a low-end 
Pentium machine. 

8 Reconstruction Error Bounds 

As shown by our experiments of the previous section, 
mining the distorted database does result in errors in 
the reconstructed support. We now provide a loose 
probabilistic bound on these errors, focusing specifi- 
cally on 1-itemsets since, as mentioned earlier, their 
errors percolate through the entire mining process. 

Consider a single column in the true matrix. Sup- 
pose, it has til's and dbsize — n 0's. We expect that 
the n l's will distort to np l's and n(l — p) 0's when 
distorted with parameter p. Similarly, we expect the 
0's to go to {dbsize — n)p 0's and (dbsize — n)(l — p) 
l's. However, note that this is assuming that "When 
we generate a random number, which is distributed as 
bernoulli(p), then the number of l's, denoted by P, in 
n trials is actually np" . But, in reality, this will not 
be so. Actually, P is distributed as binomial(n,p): 

P = binomial(n,p) 
P r {P = r,0 < r < n) = n C r p r {\ - p) n ~ r 

As the value of n increases, the sample space of 
possible outcomes expands. And the probability that 
P = np decreases. But, we are interested in an error 
bound around the value np. So, we define a, function 
P E (n,p,e) as : 

P E (n,p, e) = P r {\Number of l's - np\ < e} 

That is, P E (n,p,e) measures the probability that 
when the above experiment is conducted, (lie number 
of l's is between np — e and np + e. Clearly, 

P E (n,p, e) = XZ%-e n C rP r (l -P) n - r 

Now, let the true column have n l's and m 0's and the 
distorted column have n' l's and m' 0's. Then, given 
the distorted column. MASK reconst ructs t he support 
to 



_ = pre' _ (1 -p)m' 
U 2p-l 2p-l 

_ re' (1 —p)dbsize 

H = 2p - 1 2p- 1 ^ 

We now find the possible error in this approxima- 
tion by us ling the function P E . First, with probability 
P E (m,p, €\), the number of l's generated from the m 
O's in the initial column is between m(l — p) — ej and 
m(l -p)+e 1 . Next, with probability P E (n,p,e 2 ), 
the number of l's generated from the n l's in the ini- 
tial column is between np — < and up I ( : >. Finally, 
with probability P E (m,p,e 1 ) x P E (n,p,e- 2 ) the num- 
ber of l's in the distorted database (n') is between 
m(l -p)+np- (ei + e 2 ) and m(l -p) + np + (ei + e 2 ). 

Note that the approximation in Equation 6 is based 
on the expectation that n' is equal to m(l — p) + np. 
Hence, with probability P E (m,p, ei) x P E (n,p, e 2 ), we 
can assert that the error in the value of n' is 

Are' < ei + e 2 

Now, from Equation 6, we know that the error in re- 
construction is related to the error in the value of n' 
as follows: 

A _ Are' 
Therefore, we conclude that 



In general, let e\ = e 2 = ^-^e. Then we assert "With 
probability P E (m,p, &=le) x P E (n,p, ^e), the er- 
ror in reconstruction is less than e" . 

Note that in the above derivation, we have consid- 
ered the worst case when both the errors are in the 
same direction, that is, deviating n' from the value 
np + m{\ —p) in the same direction. This is the reason 
for adding the values of e\ and e 2 . In practice, however, 
we could expect that there is a reasonable likelihood 
that the errors follow opposite directions and therefore 
partially or fully negate each other. 

9 Related Work 

Concurrently with our work, the issue of maintaining 
privacy in association rule mining lias attracted con- 
siderable attention over the last year |l I. •">. (i. b">. 19. 
11, 8]. The problem addressed in |1 I. •">. (>. \~>\ is how to 
prevent sen sil/nn rul.t s From being inferred by the data 
miner - the proposed solutions involve either falsifying 
some of the entries in the true database or replacing 
them with NULL values. This work is complementary 
to ours since it addresses concerns about output pri- 
vacy, whereas our focus is on the privacy of the input 
data. Also note that, by definition, these techniques 



require a completely materialized true database as the 

rting point i i j pi i >p< In 

ing the data collection process itself. 

Maintaining input data privacy is considered in 
[19, 11] in the context of databases that are distributed 
across a number of sites with each site only willing to 
share data mining results, but not the source data. 
While | I9| considers data that is vertically partitioned 
(i.e., each site hosts a disjoint subset of the matrix 
columns), the complementary situation where the data 
is horizontally partitioned (i.e.. each site hosts a dis- 
joint subset of the matrix rows) is addressed in [11]. 
The solution technique in [19| requires generating and 
computing a large set of independent linear equations 

in fact , the number of equations and the number 
of terms in each equat ion is proportional to the car- 
dinality of the database. It may therefore prove to 
be expensive lor market-basket databases which typ- 
ically contain millions of customer transactions. In 
[11], on the other hand, the problem is modeled as a 
secure multi- party computation [10] and an algorithm 
that minimizes the information shared without, incur- 
ring much overhead on the mining process is presented. 
While these techniques are meaningful only in the con- 
text of distributed databases, our work is applicable to 
both centralized and distributed databases, further, 
they assume a pre-existing true database at each site, 
whereas, as mentioned earlier, our approach can pro- 
vide privacy directly at the user machine. Finally, a 
set of randomization operators for maintaining data 
privacy are presented and analyzed in [8] . 

Privacy-preserving mining in the context of classifi- 
cation rules has been investigated recently in [4, 1, 12]. 
Cryptographic protocols to ensure complete privacy in 
developing decision tree classifiers across distributed 
databases were presented in [12]. An alternative value 
distortion approach wherein a random value is added 
to each original value was taken in [4] and the privacy 
attained was quantified by the "fuzziness" provided by 
the system, that is, for a given level of confidence, the 
size of the interval that is expected to hold the origi- 
nal true value. Since we assume boolean data values, 
such an interval-based approach is not applicable in 
our context. 

It was shown in [1] that the privacy estimates of [4] 
had to be lowered when the additional knowledge that 
the miner obtains from the reconstructed aggregate 
distribution was included in the problem formulation. 
The decrease is primarily due to the fact that values 
of the distorted data may lie outside the domain of 
the original data, thereby narrowing the interval asso- 
ciated with the true value. This problem cannot occur 
in our scenario, since both the original data and the 
distorted data have exactly the same domain, namely, 
0 and 1. However, it is still possible, as explained 
very recently in [8]. that the association rules form- 
ing the mining output may be used to re-interrogate 



the distorted dal bas< id therel reduce the privacy 



10 Conclusions 

We have investigated the problem of supporting the 
conflicting goals of privacy and accuracy while min- 
ing association rules on large databases. Specifically, 
we presented a privacy metric and an analytical for- 
mula for evaluating the privacy of our MASK scheme, 
which is based on probabilistic distortion of user data, 
according to this metric. The formula showed that pri- 
vacy is a function of the sparseness of the true matrix, 
as w r ell as of the relative weight given to the privacy of 
l's as compared to O's. 

A mining process for generating frequent itemsets 
from the distorted database was also presented, along 
with a set of optimizations to address the fact that 
mining the distorted database is significantly more ex- 
pensive than mining the true database. The optimiza- 
tions significantly reduced both the number of counters 
and the amount of counting that needs to be used in 
the mining process. 

Our experimental results on synthetic and real 
databases showed that a distortion probability of p = 
0.9 (equivalently, p = 0.1) is ideally suited to provide 
both privacy and good mining results for the sparse 
market-basket type of databases that we have con- 
sidered in this study. Specifically, a privacy of over 
80% and an error of less than 10% were simultane- 
ously achieved with this setting. 

In our future work, we plan to investigate the ex- 
tension of our results to generalized [17] and quantita- 
tive [18] association rules. We also plan to refine our 
privacy estimation formulas to include the effects of us- 
ing the mining output to re-interrogate the distorted 
database. 
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ABSTRACT 

We present a framework for mining association rules from 
transactions consisting of categorical items where the data 
has been randomized to preserve privacy of individual trans- 
actions. While it is feasible to recover association rules and 
preserve privacy using a straightforward "uniform" random- 
ization, the discovered rules can unfortunately be exploited 
to find privacy breaches. We analyze the nature of privacy 
breaches and propose a class of randomization operators 
that are much more effective than uniform randomization in 
limiting the breaches. We derive formulae for an unbiased 
support estimator and its variance, which allow us to re- 
cover itemset supports from randomized datasets, and show 
how to incorporate these formulae into mining algorithms. 
Finally, we present experimental results that validate the 
algorithm by applying it on real datasets. 

1. INTRODUCTION 

The explosive progress in networking, storage, and proces- 
sor technologies is resulting in an unprecedented amount of 
digitization of information. It is estimated that the amount 
of information in the world is doubling every 20 months 
[20]. In concert with this dramatic and escalating increase 
in digital data, concerns about privacy of personal informa- 
tion have emerged globally [15] [17] [20] [24]. Privacy issues 
are further exacerbated now that the internet makes it easy 
for the new data to be automatically collected and added 
to databases [10] [13] [14] [27] [28] [29]. The concerns over 
massive collection of data are naturally extending to ana- 
lytic tools applied to data. Data mining, with its promise to 
efficiently discover valuable, non-obvious information from 
large databases, is particularly vulnerable to misuse [11] [16] 
[20] [23]. 

An interesting new direction for data mining research is 
the development of techniques that incorporate privacy con- 
cerns [3]. The following question was raised in [7]: since the 
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primary task in data mining is the development of mod- 
els about aggregated data, can we develop accurate mod- 
els without access to precise information in individual data 
records? Specifically, they studied the technical feasibility 
of building accurate classification models using training data 
in which the sensitive numeric values in a user's record have 
been randomized so that the true values cannot be estimated 
with sufficient precision. Randomization is done using the 
statistical method of value distortion [12] that returns a 
value x t + r instead of x t where r is a random value drawn 
from some distribution. They proposed a Bayesian proce- 
dure for correcting perturbed distributions and presented 
three algorithms for building accurate decision trees [9] [21] 
that rely on reconstructed distributions. 1 In [2], the au- 
thors derived an Expectation Maximization (EM) algorithm 
for reconstructing distributions and proved that the EM al- 
gorithm converged to the maximum likelihood estimate of 
the original distribution based on the perturbed data. They 
also pointed out that the EM algorithm was in fact identical 
to the Bayesian reconstruction procedure in [7], except for 
an approximation (partitioning values into intervals) that 
was made by the latter. 

1.1 Contributions of this Paper 

We continue the investigation of the use of randomization 
in developing privacy-preserving data mining techniques, and 
extend this line of inquiry along two dimensions: 

• categorical data instead of numerical data, and 

• association rule mining [4] instead of classification. 
We will focus on the task of finding frequent itemsets in 
association rule mining, which we briefly review next. 

Definition 1. Suppose we have a set I of n items: T = 
{oi,o 2 ,... ,o n }. Let T be a sequence of N transactions 
T = (ti, t 2 , ■ ■ ■ , tjv) where each transaction t t is a subset of 
X. Given an itemset A C 2T, its support supp T (^4) is defined 



supp T (v4) : 



_ #{teT\ACt} 



An itemset A C I is called frequent in T if supp (A) J> t, 
where t is a user-defined parameter. 



We consider the following setting. Suppose we have a 
>erver and many clients. Each client has a set of items (e.g., 



x Once we have reconstructed distributions, it is straightfor- 
ward to build classifiers that assume independence between 
attributes, such as Naive Bayes [19] . 



books or web pages or TV programs). The clients want the 
server to gather statistical information about associations 
among items, perhaps in order to provide recommendations 
to the clients. However, the clients do not want the server 
to know with certainty who has got which items. When a 
client sends its set of items to the server, it modifies the 
set according to some specific randomization policy. The 
server then gathers statistical information from the modified 
sets of items (transactions) and recovers from it the actual 
associations. 

The following are the important results contained in this 

• In Section 2, we show that a straightforward uniform 
randomization leads to privacy breaches. 

• We formally model and define privacy breaches in Sec- 

• We present a class of randomization operators in Sec- 
tion 4 that can be tuned for different tradeoffs between 
discoverability and privacy breaches. We derive for- 
mulae for the effect of randomization on support, and 
show how to recover the original support of an associ- 
ation from the randomized data. 

• We present experimental results on two real datasets 
in Section 5, as well as graphs showing the relationship 
between discoverability, privacy, and data characteris- 

1.2 Related Work 

There has been extensive research in the area of statistical 
databases motivated by the desire to provide statistical in- 
formation (sum, count, average, maximum, minimum, pth 
percentile, etc.) without compromising sensitive informa- 
tion about individuals (see surveys in [1] [22].) The pro- 
posed techniques can be broadly classified into query re- 
striction and data perturbation. The query restriction fam- 
ily includes restricting the size of query result, controlling 
the overlap amongst successive queries, keeping audit trail 
of all answered queries and constantly checking for possi- 
ble compromise, suppression of data cells of small size, and 
clustering entities into mutually exclusive atomic popula- 
tions. The perturbation family includes swapping values 
between records, replacing the original database by a sam- 
ple from the same distribution, adding noise to the values 
in the database, adding noise to the results of a query, and 
sampling the result of a query. There are negative results 
showing that the proposed techniques cannot satisfy the con- 
flicting objectives of providing high quality statistics and at 
the same time prevent exact or partial disclosure of individ- 
ual information [1]. 

The most relevant work from the statistical database lit- 
erature is the work by Warner [26], where he developed 
the "randomized response" method for survey results. The 
method deals with a single boolean attribute (e.g., drug ad- 
diction). The value of the attribute is retained with prob- 
ability p and flipped with probability 1 -&p. Warner then 
derived equations for estimating the true value of queries 
such as COUNT (Age = 42 & Drug Addiction = Yes). The 
approach we present in Section 2 can be viewed as a gener- 
alization of Warner's idea. 

Another related work is [25], where they consider the 
problem of mining association rules over data that is ver- 
tically partitioned across two sources, i.e, for each transac- 
tion, some of the items are in one source, and the rest in the 



other source. They use multi-party computation techniques 
for scalar products to be able to compute the support of an 
itemset (when the two subsets that together form the item- 
set are in different sources), without either source revealing 
exactly which transactions support a subset of the itemset. 
In contrast, we focus on preserving privacy when the data 
is horizontally partitioned, i.e., we want to preserve privacy 
for individual transactions, rather than between two data 
sources that each have a vertical slice. 

Related, but not directly relevant to our current work, 
is the problem of inducing decision trees over horizontally 
partitioned training data originating from sources who do 
not trust each other. In [16], each source first builds a lo- 
cal decision tree over its true data, and then swaps values 
amongst records in a leaf node of the tree to generate ran- 
domized training data. Another approach, presented in [18], 
does not use randomization, but makes use of cryptographic 
oblivious functions during tree construction to preserve pri- 
vacy of two data sources. 

2. UNIFORM RANDOMIZATION 

A straightforward approach for randomizing transactions 
would be to generalize Warner's "randomized response" me- 
thod, described in Section 1.2. Before sending a transaction 
to the server, the client takes each item and with probabil- 
ity p replaces it by a new item not originally present in this 
transaction. Let us call this process uniform randomization. 

Estimating true (nonrandomized) support of an itemset 
is nontrivial even for uniform randomization. Randomized 
support of, say, a 3-itemset depends not only on its true 
support, but also on the supports of its subsets. Indeed, it 
is much more likely that only one or two of the items are 
inserted by chance than all three. So, almost all "false" oc- 
currences of the itemset are due to (and depend on) high 
subset supports. This requires estimating the supports of 
all subsets simultaneously. (The algorithm is similar to the 
algorithm presented in Section 4 for select-a-size random- 
ization, and the formulae from Statements 1, 3 and 4 apply 
here as well.) For large values of p, most of the items in 
most randomized transactions will be "false" , so we seem to 
have obtained a reasonable privacy protection. Also, if there 
are enough clients and transactions, then frequent itemsets 
will still be "visible", though less frequent than originally. 
For instance, after uniform randomization with p = 80%, an 
itemset of 3 items that originally occurred in 1% transac- 
tions will occur in about 1% • (0.2) 3 = 0.008% transactions, 
which is about 80 transactions per each million. The op- 
posite effect of "false" itemsets becoming more frequent is 
comparatively negligible if there are many possible items: 
for 10,000 items, the probability that, say, 10 randomly in- 
serted items contain a given 3-itemset is less than 10 _7 %. 

Unfortunately, this randomization has a problem. If we 
know that our 3-itemset escapes randomization in 80 per 
million transactions, and that it is unlikely to occur even 
once because of randomization, then every time we see it 
in a randomized transaction we know with near certainty of 
its presence in the nonrandomized transaction. With even 
more certainty we will know that at least one item from this 
itemset is "true" : as we have mentioned, a chance insertion 
of only one or two of the items is much more likely than 
of all three. In this case we can say that a privacy breach 
has occurred. Although privacy is preserved on average, 
personal information leaks through uniform randomization 



for some fraction of transactions, despite the high value of p. 

The rest of the paper is devoted to defining a framework 
for studying privacy breaches and developing techniques for 
finding frequent itemsets while avoiding breaches. 

3. PRIVACY BREACHES 

Definition 2. Let (Q,^ 7 , P) be a probability space of el- 
ementary events over some set Q and cr-algebra T . A ran- 
domization operator is a measurable function 

R : Q x {all possible T} -> {all possible T} 

that randomly transforms a sequence of N transactions into 
a (usually) different sequence of N transactions. Given a 
sequence of N transactions T, we shall write T' = R(T), 
where T is constant and R(T) is a random variable. 

Definition 3. Suppose that a nonrandomized sequence T 
is drawn from some known distribution, and t t £ T is the 
i-th transaction in T. A general privacy breach of level p 
with respect to a property P(t t ) occurs if 

3T' : P [P(ti) | R(T) = T'] > p. 

We say that a property Q(T') causes a privacy breach of 
level p with respect to P(U) if 

P[P(t,)| Q(R(T))]>p. 

When we define privacy breaches, we think of the prior 
distribution of transactions as known, so that it makes sense 
to speak about a posterior probability of a property P(U) 
versus prior. In practice, however, we do not know the 
prior distribution. In fact, there is no prior distribution; the 
transactions are not randomly generated. However, model- 
ing transactions as being randomly generated from a prior 
distribution allows us to cleanly define privacy breaches. 

Consider a situation when, for some transaction t % £ T, 
an itemset ACT and an item a £ A, the property "A C t ■ £ 
T'" causes a privacy breach w. r. t. the property "a £ In 
other words, the presence of A in a randomized transaction 
makes it likely that item a is present in the corresponding 
nonrandomized transaction. 

Definition 4- We say that itemset A causes a privacy 
breach of level p if for some item a £ A and some i £ 1 . . . N 
we have P [a £ U | A C i-] ^ p. 

We will focus on controlling the class of privacy breaches 
given by Definition 4. Thus we ignore the effect of other 
information the server obtains from a randomized transac- 
tion, such as which items the randomized transaction does 
not contain, or the randomized transaction size. We also 
do not attempt to control breaches that occur because the 
server knows some other information about items and clients 
besides the transactions. For example, the server may know 
some geographical or demographic data about the clients. 
Finally, in Definition 4, we only considered positive breaches, 
i.e., we know with high probability that an item was present 
in the original transaction. In some scenarios, being confi- 
dent that an item was not present in the original transaction 
may also be considered a privacy breach. 



4. ALGORITHM 

"Where does a wise man hide a leaf? In the 
forest. But what does he do if there is no forest?" 
... "He grows a forest to hide it in." - G.K. 
Chesterton, "The Sign of the Broken Sword" 

The intuition of breach control is quite simple: in addition to 
replacing some of the items, we shall insert so many "false" 
items into a transaction that one is as likely to see a "false" 
itemset as a "true" one. 

4.1 Randomization Operators 

Definition 5. We call randomization R a per-transaction 
randomization if, for T = (ti, ti, . . . , tjv), we can represent 
R(T) as 

R(ti, t 2 , . . . , t N ) = h), R(2, t 2 ), . . . , R(N, t N )), 

where R(i, t) are independent random variables whose dis- 
tributions depend only on t (and not on i). We shall write 
t' i = R{i,t i ) = R{t i ). 

Definition 6. A randomization operator R is called item- 
invariant if, for every transaction sequence T and for ev- 
ery permutation 7r : 1 — )■ 1 of items, the distribution of 
7r -1 .R(7rT) is the same as of R(T). Here ttT means the ap- 
plication of 7r to all items in all transactions of T at once. 

Definition 7. A select-a-size randomization operator has 
the following parameters, for each possible input transaction 

• Default probability of an item (also called randomiza- 
tion level) pm £ (0, 1); 

• Transaction subset size selection probabilities p m [0], 
p m [l], • • • ,Pm[m], such that every p m [j] > 0 and 

Pm[0]+ P m[l] + ...+Pm[m] = l. 

Given a sequence of transactions T = (ti , t2 , • • • ,tiv), the 
operator takes each transaction t % independently and pro- 
ceeds as follows to obtain transaction t[ (m = |t,|). 

1. The operator selects an integer j at random from the 
set {0, 1, . . . , m} so that P [j is selected] = p m [j]. 

2. It selects j items from t t , uniformly at random (with- 
out replacement). These items, and no other items 
of U, are placed into t' % . 

3. It considers each item a 0 t % in turn and tosses a coin 
with probability p m of "heads" and 1 <S>p m of "tails" . 
All those items for which the coin faces "heads" are 
added to t' % . 

Remark 1. Both uniform (Section 2) and select-a-size op- 
erators are per-transaction because they apply the same 
randomization algorithm to each transaction independently. 
They are also item-invariant since they do not use any item- 
specific information (if we rename or reorder the items, the 
outcome probabilities will not be affected). 



Definition 8. A cut-and-paste randomization operator is 
a special case of a select-a-size operator (and which we shall 
actually test on datasets). For each possible input transac- 
tion size m, it has two parameters: p m £ (0, 1) (randomiza- 
tion level) and an integer K m > 0 (the cutoff). The operator 
takes each input transaction t t independently and proceeds 
as follows to obtain transaction t[ (here m = \t t \) : 

1. It chooses an integer j uniformly at random between 
0 and K m ; if J > m, it sets j = m. 

2. The operator selects j items out of t, uniformly at ran- 
dom (without replacement). These items are placed 

3. Each other item (including the rest of t t ) is placed 
into t' % with probability p m , independently. 

Remark 2. For any m, a cut-and-paste operator has only 
two parameters, p m and K m , to play with; moreover, Km 
is an integer. Because it is easy to find optimal values for 
these parameters (Section 4.4), we chose to test this opera- 
tor, leaving open the problem of optimizing the m parame- 
ters of the "unabridged" select-a-size. To see that cut-and- 
paste is a case of select-a-size, let us write down the formulae 
for the Pm[i]'s: 

vM = "g" > (7 «>--■ 

f 1 &m/(K + 1) if i = m and i < K 
' \ 1/(K + 1) otherwise 

Now let us give one example of a randomization operator 
that is not a per-transaction randomization, because it uses 
the knowledge of several transactions per each randomized 
transaction. 

Example 1. The mixing randomization operator has one 
integer parameter K ^ 2 and one real- valued parameter p £ 
(0, 1). Given a sequence of transactions T = (ti , t 2 , ■ ■ ■ , tjv), 
the operator takes each transaction t % independently and 
proceeds as follows to obtain transaction tj: 

1. Other than U, pick K&L more transactions (with re- 
placement) from T and union the K transactions as 
sets of items. Let t" be this union. 

2. Consider each item a £ t" in turn and toss a coin with 
probability p of "heads" and 1 Op of "tails" . 

3. All those items for which the coin faces "tails" are 
removed from the transaction. The remaining items 
constitute the randomized transaction. 

For the purpose of privacy-preserving data mining, it is 
natural to focus mostly on per-transaction randomizations, 
since they are the easiest and safest to implement. Indeed, 
a per-transaction randomization does not require the users 
(who submit randomized transactions to the server) to com- 
municate with each other in any way, nor to exchange ran- 
dom bits. On the contrary, implementing mixing random- 
ization, for example, requires to organize an exchange of 
nonrandomized transactions between users, which opens an 
opportunity for cheating or eavesdropping. 



4.2 Effect of Randomization on Support 

Let T be a sequence of transactions of length N, and 
let A be some subset of items (that is, ACT). Suppose 
we randomize T and get T' = R(T). The support s' = 
supp T (A) of A for T' is a random variable that depends 
on the outcome of randomization. Here we are going to 
determine the distribution of s' , under the assumption of 
having a per-transaction and item-invariant randomization. 

Definition 9. The fraction of the transactions in T that 
have intersection with A of size I among all transactions in 
T is called partial support of A for intersection size I: 

WM):- * <t6T| * Mn "-' 1 . w 

It is easy to see that supp T (^4) = supp^(j4) for k = \A\, 
and that 

^suppf(^l) = 1 

since those transactions in T that do not intersect A at all 
are covered in suppj(j4). 

Definition 10. Suppose that our randomization operator 
is both per-transaction and item-invariant. Consider a trans- 
action t of size m and an itemset A C T of size k. After 
randomization, transaction t becomes t' . We define 

P? [I -> I'] = P [I -> I'] := 

p[#(t'nA) = i'\#(tnA) = i]. (3) 

Here both I and V must be integers in {0, 1, . . . , k}. 

Remark 3. The value of P% [I -t V] is well-defined (does 
not depend on any other information about t and A, or other 
transactions in T and T' besides t and t'). Indeed, because 
we have a per-transaction randomization, the distribution of 
t' depends neither on other transactions in T besides t, nor 
on their randomized outcomes. If there were other t\ and B 
with the same (m, k, I), but a different probability (3) for the 
same I , we could consider a permutation n of I such that 
wt = t\ and nA = B; the application of 7r or of 7r _1 would 
preserve intersection sizes I and I'. By item-invariance we 

p [#(*' n A) = i'] = p [#(7r-\R(7rt) nA) = i'], 

but by the choice of ir we also have 

p tfOr-^Ort) n A) = I'] = p m^-'Rih) n 7r-\B) = I'] 
= p[#(t[nB) = i'] + p[#(t'nA) = i'], 

a contradiction. 

STATEMENT 1. Suppose that our randomization operator 
is both per-transaction and item-invariant. Suppose also that 
all the N transactions in T have the same size m. Then, for 
a given subset ACT, \A\ = k, the random vector 

N ■ (s' 0 , s[, . . . , s' k ), where s'i := suppf' (A) (4) 

is a sum of k + 1 independent random vectors, each having 
a multinomial distribution. Rs expected value is given by 

E (so, s[ , . . . , s' k ) T = P ■ (s 0 , 5i , • • • , s k ) T (5) 



where P is the {k + 1) x (fc + 1) matrix with elements Pi* ; = 
P[i-> I'], and the covariance matrix is given by 

Cov( S ' 0 , S [,..., S ' k ) T = ±-j2s lD [l] (6) 

where each D[l] is a (k + 1) x (k + 1) matrix with elements 

D[l] t3 =P[l^i]- S t=3 &p[l^i].p[l^ j] . ( 7 ) 

Here si denotes suppJ"(A), and the T over vectors denotes 
the transpose operation; S Z=J is one if i = j and zero other- 

PROOF. See Appendix A.l. □ 

Remark 4- In Statement 1 we have assumed that all trans- 
actions in T have the same size. If this is not so, we have 
to consider each transaction size separately and then use 
per-transaction independence. 

STATEMENT 2. For a select-a-size randomization with 
randomization level p and size selection probabilities 
{Pm[j}}, we have: 

3 = 0 g =max{0,;, + I-m,I+I'-fc} / "M 

PROOF. See Appendix A. 2. □ 

4.3 Support Recovery 

Let us assume that all transactions in T have the same 
size ra, and let us denote 

s := (so,s 1: ... ,s k ) T , s' := (s 0 ,s 1: ... ,s k ) T ; 

then, according to (5), we have 

Ej' = P-j. (9) 

Denote Q = P _1 (assume that it exists) and multiply both 
sides of (9) by Q: 

s = Q-Es' = EQs'. 

We have thus obtained an unbiased estimator for the original 
partial supports given randomized partial supports: 

At := Q ■ S' (10) 
Using (6), we can compute the covariance matrix of s est : 

Cov = Cov (Q • s') = Q (Cov s') Q T = 

= ± r Y,s l QD[l]Q T . (11) 

If we want to estimate this covariance matrix by look- 
ing only at randomized data, we may use <f est instead of s 
in (11): 

(Covs^u = ±-J2(^)iQD[i]Q T - 



This estimator is also unbiased: 

E (Cov r. rt ). rt = ± ■ f^(E Q D[l] Q T = Cov f. rt . 

In practice, we want only the fc-th coordinate of J*, that is, 
the support s = supp T (^4) of our itemset A in T. We denote 
by s the fc-th coordinate of i* es t, and use s to estimate s. 
Let us compute simple formulae for s, its variance and the 
unbiased estimator of its variance. Denote 

9[l«-r] :=Q U >. 

Statement 3. 
(Var S) M( = ^ E *£' ( 9 [ fc Z 'l 2 ^ 9 [ fc «" Z 'l )• 



PROOF. See Appendix A. 3. □ 

We conclude this subsection by giving a linear coordinate 
transformation in which the matrix P from Statement 1 
becomes triangular. (We use this transformation for privacy 
breach analysis in Section 4.4.) The coordinates after the 
transformation have a combinatorial meaning, as given in 
the following definition. 

Definition 11. Suppose we have a transaction sequence T 
and an itemset A C X. Given an integer I between 0 and 
k = \A\, consider all subsets C C A of size I. The sum of 
supports of all these subsets is called the cumulative support 
for A of order I and is denoted as follows: 

S, = E,(j4,T) := SU PP T (C). 

CCA, |C| = ! 

E := (Eo.Ei,... ,S/=) T (12) 

STATEMENT 4. The vector E of cumulative supports is 
a linear transformation of the vector s of partial supports, 
namely, 

s < = E (])^ and s < = EMr'(;) s >i (13) 

in the E and E' space (instead of s and s' ) matrix P is 
lower triangular. 

PROOF. See Appendix A. 4. □ 

4.4 Limiting Privacy Breaches 

Here we determine how privacy depends on randomiza- 
tion. We shall use Definition 4 and assume a per-transaction 
and item-invariant randomization. 

Consider some itemset ACT and some item a 6 A; fix 
a transaction size m. We shall assume that m is known to 
the server, so that we do not have to combine probabilities 



for different nonrandomized sizes. Assume also that a par- 
tial support Si = suppf(j4) approximates the corresponding 
prior probability P [#(t Pi ^4) = I]. Suppose we know the 
following prior probabilities: 

s+ := P[#(tf)A)=l, aet], 

5f := P[#(tnA)=l, a#t]. 

Notice that si = sf + sj~ simply because 



equally frequently as t Pi A: 



a e t & #(tnj4) = i, c 

a $t & #(tnA) =1. 



Let us use these priors and compute the posterior proba- 
bility of a G t given A C t': 

= ^P[#(tni4) = I, aet, Act']/ *i-pP->*] 
= f P[#(tni) = Uet].pMi]/^ i .p[Ui] 

Thus, in order to prevent privacy breaches of level 50% as 
defined in Definition 4, we need to ensure that always 

X>+-P[*^fc] < 0.6-X>-Pp->*]. (W) 

The problem is that we have to randomize the data before 
we know any supports. Also, we may not have the luxury of 
setting "oversafe" randomization parameters because then 
we may not have enough data to perform a reasonably ac- 
curate support recovery. One way to achieve a compromise 
is to: 



1. Estimate maximum possible support s max (fc,m) of a 
fc-itemset in the transactions of given size m, for dif- 
ferent fc and m; 



2. Given the maximum supports, find values for si and 
Sj that are most likely to cause a privacy breach; 

3. Make randomization just strong enough to prevent 
such a privacy breach. 

Since sj = 0, the most privacy-challenging situations occur 
when so is small, that is, when our itemset A and its subsets 
are frequent. 

In our experiments we consider a privacy-challenging fc- 
itemset A such that, for every I > 0, all its subsets of size I 
have the maximum possible support Smax{l, m). The partial 
supports for such a test-itemset are computed from the cu- 
mulative supports Hi using Statement 4. By it and by (12), 
we have (I > 0) 



(15) 



= P[#(tnj4) = j, aet] = 

= P[aet\#(tf)A)=l]- Sl = i/k ■ 



(16) 



While c 



i construct c 



s that 8 



nore privacy- 
challenging (for example, if a 6 A occurs in a transaction 
every time any nonempty subset of A does), we found the 
above model (15) and (16) to be sufficiently pessimistic on 
our datasets. 

We can now use these formulae to obtain cut-and-paste 
randomization parameters p m and K m as follows. Given m, 
consider all cutoffs from K m = 3 to some /C ma x (usually this 
i^max equals the maximum transaction size) and determine 
the smallest randomization levels p m {Km) that satisfy (14). 
Then select (Km, Pm) that gives the best discoverability (by 
computing the lowest discoverable supports, see Section 5.1). 

4.5 Discovering Associations 

We show how to discover itemsets with high true support 
given a set of randomized transactions. Although we use the 
Apriori algorithm [5] to make the ideas concrete, the mod- 
ifications directly apply to any algorithm that uses Apriori 
candidate generation, i.e., to most current association dis- 
covery algorithms. 2 The key lattice property of supports 
used by Apriori is that, for any two itemsets A C B, the 
true support of A is equal to or larger than the true support 
of B. A simplified version of Apriori, given a (nonrandom- 
ized) transactions file and a minimum support s m in, works 
as follows: 

1. Let k = 1, let "candidate sets" be all single items. 
Repeat the following until no candidate sets are left: 



(b) Discard all candidate sets whose support is be- 

(c) Save the remaining candidate sets for output; 

(d) Form all possible (fc + l)-itemsets such that all 
their fc-subsets are among the remaining candi- 
dates. Let these itemsets be the new candidate 

(e) Let fc = fc + 1. 

2. Output all the saved itemsets. 

It is (conceptually) straightforward to modify this algo- 
rithm so that now it reads the randomized dataset, computes 
partial supports of all candidate sets (for all nonrandomized 
transaction sizes) and recovers their predicted supports and 
sigmas using the formulae from Statement 3. However, for 
the predicted supports the lattice property is no longer true. 
It is quite likely that for an itemset that is slightly above 
minimum support and whose predicted support is also above 
minimum support, that one of its subsets will have predicted 
support below minimum support. So if we discard all candi- 
dates below minimum support for the purpose of candidate 
generation, we will miss many (perhaps even the majority) 



2 The r 



since there are ( fc ) j-subsets in A. The values of s~f follow if 
we note that all Z-subsets of A, with a and without, appear 



class of algorithms where this would not apply 
' ■ < ■ ' , [8]. 



e those that find only maximal frequent itemsets. „ , L j 
However, randomization precludes finding very long item- 
sets, so this is a moot point. 



of the longer frequent itemsets. Hence, for candidate gen- 
eration, we discard only those candidates whose predicted 
support is "significantly" smaller than s m i n , where signifi- 
cance is measured by means of predicted sigmas. Here is 
the modified version of Apriori: 

1. Let k = 1, let "candidate sets" be all single-item sets. 
Repeat the following until k is too large for support 
recovery (or until no candidate sets are left): 

(a) Read the randomized data file and compute the 
partial supports of all candidate sets, separately 
for each nonrandomized transaction size 3 ; 

(b) Recover the predicted supports and sigmas for the 
candidate sets; 

(c) Discard every candidate set whose support is be- 
low its candidate limit; 

(d) Save for output only those candidate sets whose 
predicted support is at least s m i n ; 

(e) Form all possible (fc + l)-itemsets such that all 
their fc-subsets are among the remaining candi- 
dates. Let these itemsets be the new candidate 

(f) Let k = k + l. 

2. Output all the saved itemsets. 

We tried s m ; n and s m ; n <£>2<t as the candidate limit, 
and found that the former does a little better than the latter. 
It prunes more itemsets and therefore makes the algorithm 
work faster, and, when it discards a subset of an itemset 
with high predicted support, it usually turns out that the 
true support of this itemset is not as high. 

5. EXPERIMENTAL RESULTS 

Before we come to the experiments with datasets, we first 
show in Section 5.1 how our ability to recover supports de- 
pends on the permitted breach level, as well as other data 
characteristics. We then describe the real-life datasets in 
Section 5.2, and present results on these datasets in Sec- 
tion 5.3. 

5.1 Privacy Discoverability and Dataset Char- 
acteristics 

We define the lowest discoverable support as the support 
at which the predicted support of an itemset is four sigmas 
away from zero, i.e, we can clearly distinguish the support 
of this itemset from zero. In practice, we may achieve rea- 
sonably good results even if the minimum support level is 
slightly lower than four sigma (as was the case for 3-itemsets 
in the randomized soccer, see below). However, the lowest 
discoverable support is a nice way to illustrate the interac- 
tion between discoverability, privacy breach levels, and data 
characteristics. 

Figure 1 shows how the lowest discoverable support changes 
with the privacy breach level. For higher privacy breach 
levels such as 95% (which could be considered a "plausi- 
ble denial" breach level), we can discover 3-itemsets at very 
low supports. For more conservative privacy breach levels 

3 In our experiments, the nonrandomized transaction size is 
always known and included as a field into every randomized 
transaction 




Privacy breach level, % 



Figure 1: Lowest discoverable support for differ- 
ent breach levels. Transaction size is 5, five million 
transactions. 




of Transactions, n 



Figure 2: Lowest discoverable support versus num- 
ber of transactions. Transaction size is 5, breach 
level is 50%. 



such as 50%, the lowest discoverable support is significantly 
higher. It is interesting to note that at higher breach lev- 
els (i.e. weaker randomization) it gets harder to discover 
1-itemset supports than 3-itemset supports. This happens 
because the variance of a 3-itemset predictor depends highly 
nonlinearly on the amount of false items added while ran- 
domizing. When we add fewer false items at higher breach 
levels, we generate so much fewer false 3-itemset positives 
than false 1-itemset positives that 3-itemsets get an advan- 
tage over single items. 

Figure 2 shows that the lowest discoverable support is 
roughly inversely proportional to the square root of the num- 
ber of transactions. Indeed, the lowest discoverable sup- 
port is defined to be proportional to the standard deviation 
(square root of the variance) of this support's prediction. If 
all the partial supports are fixed, the prediction's variance 
is inversely proportional to the number N of transactions 
according to Statement 3. In our case, the partial supports 
depend on N (because the lowest discoverable support does), 
i.e. they are not fixed; however, this does not appear to af- 
fect the variance very significantly (but justifies the word 
"roughly" ) . 

Finally, Figure 3 shows that transaction size has a sig- 




Figure 3: Lowest discoverable support for different 
transaction sizes. Five million transactions, breach 
level is 50%. 



nificant influence on support discoverability. In fact, for 
transactions of size 10 and longer, it is typically not possi- 
ble to make them both breach-safe and simultaneously get 
useful information for mining transactions. Intuitively, a 
long transaction contains too much personal information to 
hide, because it may contain long frequent itemsets whose 
appearance in the randomized transaction could result in a 
privacy breach. We have to insert a lot of false items and cut 
off many true ones to ensure that such a long itemset in the 
randomized transaction is about as likely to be a false pos- 
itive as to be a true positive. Such a strong randomization 
causes an exceedingly high variance in the support predictor 
for 2- and especially 3-itemsets, since it drives down their 
probability to "tunnel" through while raising high the prob- 
ability of a false positive. In both our datasets we discard 
long transactions. The question of how to safely randomize 
and mine long transactions is left open. 

5.2 The Datasets 

We experimented with two "real-life" datasets. The soccer 
dataset is generated from the clickstream log of the 1998 
World Cup Web site, which is publicly available at 
ftp : //researchsmp2 . cc . vt . edu/pub/worldcup/ 4 . We scan- 
ned the log and produced a transaction file, where each 
transaction is a session of access to the site by a client. Each 
item in the transaction is a web request. Not all web requests 
were turned into items; to become an item, the request must 
satisfy the following: 

1. Client's request method is GET; 

2. Request status is OK; 

3. File type is HTML. 

A session starts with a request that satisfies the above prop- 
erties, and ends when the last click from this ClientID time- 
outs. The timeout is set as 30 minutes. All requests in a ses- 
sion have the same ClientID. The soccer transaction file was 
then processed further: we deleted from all transactions the 
items corresponding to the French and English front page 
frames, and then we deleted all empty transactions and all 
transactions of size above 10. The resulting soccer dataset 

4 M. Arlitt and T. Jin, "1998 World Cup Web 
Site Access Logs", August 1998. Available at 

http : / / www . acm . org/sigcomm/ITA/ 



Figure 4: Number of transactions for each transac- 
tion size in the soccer and mailorder datasets. 



consists of 6,525,879 transactions, distributed as shown in 
Fig. 4. 

The mailorder dataset is the same as that used in [6]. 
The original dataset consisted of around 2.9 million transac- 
tions, 15,836 items, and around 2.62 items per transaction. 
Each transaction was the set of items purchased in a single 
mail order. However, very few itemsets had reasonably high 
supports. For instance, there were only two 2-itemsets with 
support ^0.2%, only five 3-itemsets with support ^0.05%. 
Hence we decided to substitute all items by their parents in 
the taxonomy, which had reduced the number of items from 
15836 to 96. It seems that, in general, moving items up 
the taxonomy is a natural thing to do for preserving privacy 
without losing aggregate information. We also discarded all 
transactions of size ^ 8 (which was less than 1% of all trans- 
actions) and finally obtained a dataset containing 2, 859, 314 
transactions (Fig. 4). 

5.3 The Results 

We report the results for both datasets at a minimum 
support that is close to the lowest discoverable support, in 
order to show the resilience of our algorithm even at these 
very low support levels. We targeted a conservative breach 
level of 50%, so that, given a randomized transaction, for any 
item in the transaction it is at least as likely that someone 
did not buy that item (or access a web page) as that they 
did buy that item. 

We used cut-and-paste randomization (see Definition 8) 
that has only two parameters, randomization level and cut- 
off, per each transaction size. We chose a cutoff of 7 for 
our experiments as a good compromise between privacy and 
discoverability. Given the values of maximum supports, we 
then used the methodology from Section 4.4 to find the low- 
est randomization level such that the breach probability (for 
each itemset size) is still below the desired breach level. The 
actual parameters (Km is the cutoff, p m is the randomiza- 
tion level for transaction size m) for soccer were: 



Krn 



10 



' 16.8 21.4 32.2 35.3 42.9 46.1 42.0 40.9 39.5 



and for mailorder w 



8.9 20.4 25.0 33.4 43.5 50.5 E 



Table 1 shows what happens if we mine itemsets from both 
randomized and nonrandomized files and then compare the 
results. We can see that, even for a low minimum support 
of 0.2%, most of the itemsets are mined correctly from the 
randomized file. There are comparatively few false posi- 
tives (itemsets wrongly included into the output) and even 
fewer false drops (itemsets wrongly omitted). The predicted 
sigma for 3-itemsets ranges in 0.066<S0.07% for soccer and 
in 0.047<S0.048% for mailorder; for 2- and 1-itemsets sigmas 

One might be concerned about the true supports of the 
false positives. Since we know that there are many more 
low-supported itemsets than there are highly supported, we 
might wonder whether most of the false positives are out- 
liers, that is, have true support near zero. We have indeed 
seen outliers; however, it turns out that most of the false 
positives are not so far off. The tables 2 and 3 show that 
usually the true supports of false positives, as well as the 
predicted supports of false drops, are closer to 0.2% than to 
zero. This good news demonstrates the promise of random- 
ization as a practical privacy-preserving approach. 
Privacy Analysis We evaluate privacy breaches, i.e., the 
conditional probabilities from Definition 4, as follows. We 
count the occurrences of an itemset in a randomized transac- 
tion and its sub-items in the corresponding nonrandomized 
transaction. For example, assume an itemset {a, 6, c} oc- 
curs 100 times in the randomized data among transactions 
of length 5. Out of these 100 occurrences, 60 of the corre- 
sponding original transactions had the item b. We then say 
that this itemset caused a 60% privacy breach for transac- 
tions of length 5, since for these 100 randomized transac- 
tions, we estimate with 60% confidence that the item b was 
present in the original transaction. 

Out of all sub-items of an itemset, we choose the item that 
causes the worst privacy breach. Then, for each combination 
of transaction size and itemset size, we compute over all 
frequent 5 itemsets the worst and the average value of this 
breach level. Finally, we pick the itemset size that gave the 
worst value for each of these two values. 

Table 4 shows the results of the above analysis. To the left 
of the semicolon is the itemset size that was the worst. For 
instance, for all transactions of length 5 for soccer, the worst 
average breach was with 4-itemsets (43.9% breach), and the 
worst breach was with a 5-itemset (49.7% breach). We can 
see that, apart from fluctuations, the 50% level is observed 
everywhere except of a little "slip" for 9- and 10-item trans- 
actions of soccer. The "slip" resulted from our decision 
to use the corresponding maximal support information only 
for itemset sizes up to 7 (while computing randomization 
parameters). 6 However, since such long associations cannot 
be discovered, in practice, we will not get privacy breaches 
above 50%. 

Summary Despite choosing a conservative privacy breach 
level of 50%, and further choosing a minimum support around 
the lowest discoverable support, we were able to successfully 
find most of the frequent itemsets, with relatively small num- 
bers of false drops and false positives. 
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Table 1: Results on Real Datasets 
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Table 2: Analysis of false drops 
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5 If there are no frequent itemsets for some combination, we Table 3: Analysis of false positives 

pick the itemsets with the highest support. 

6 While we could have easily corrected the slip, we felt it 

more instructive to leave it in. 



Transaction size: 


1 


2 


3 


4 


5 6 




8 


9 


10 


Worst Average: 
Worst of the Worst: 


1: 4.4% 
1: 45.5% 


2: 20.2% 
2: 45.4% 


3: 39.2% 
3: 53.2% 


4: 44.5% 
4: 49.8% 


4: 43.9% 4: 37.5% 
5: 49.7% 5: 42.7% 


4: 36.2% 
5: 41.8% 


4: 38.7% 
5: 44.5% 


8: 51.0% 
9: 66.2% 


10: 49.4% 
10: 65.6% 
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Transaction size: 


1 


2 


3 


4 5 


6 


7 


Worst Average: 
Worst of the Worst: 


1: 12.0% 
1: 47.6% 


2: 27.5% 
2: 51.9% 


3: 48.4% 
3: 53.6% 


4: 51.5% 5: 51.7% 
4: 53.1% 5: 53.6% 


5: 51.9% 
6: 55.4% 


6: 49.8% 
7: 51.9% 



Table 4: Actual Privacy Breaches 



6. CONCLUSIONS 

In this paper, we have presented three key contributions 
toward mining association rules while preserving privacy. 
First, we pointed out the problem of privacy breaches, pre- 
sented their formal definitions and proposed a natural solu- 
tion. Second, we gave a sound mathematical treatment for a 
class of randomization algorithms and derived formulae for 
support and variance prediction, and showed how to incor- 
porate these formulae into mining algorithms. Finally, we 
presented experimental results that validated the algorithm 
in practice by applying it to two real datasets from different 
domains. 

We conclude by raising three interesting questions for fu- 
ture research. Our approach deals with a restricted (albeit 
important) class of privacy breaches; can we extend it to 
cover other kinds of breaches? Second, what are the theo- 
retical limits on discoverability for a given level of privacy 
(and vice versa)? Finally, can we combine randomization 
and cryptographic protocols to get the strengths of both 
without the weaknesses of either? 
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APPENDIX 
A. PROOFS 

A.l Proof of Statement 1 



Proof. Each coordinate N ■ s[, of the vector in (4) is, 
by definition of partial supports, just the number of trans- 
actions in the randomized sequence T" that have intersec- 
tions with A of size I' . Each randomized transaction t' con- 
tributes to one and only one coordinate N ■ s' llt namely to 
the one with I 1 = #(*' n A). Since we are dealing with a 
per-transaction randomization, different randomized trans- 
actions contribute independently to one of the coordinates. 
Moreover, by item-invariance assumption, the probability 
that a given randomized transaction contributes to the co- 
ordinate number I' depends only on the size of the original 
transaction t (which equals m) and the size I of intersection 
tnA. This probability equals p[l -> I']. 

So, for all transactions in T that have intersections with A 
of the same size I (and there are N ■ si such transactions) the 
probabilities of contributing to various coordinates N-s' v are 
the same. We can split all N transactions into k + 1 groups 
according to their intersection size with A. Each group con- 
tributes to the vector in (4) as a multinomial distribution 
with probabilities 

(pp-*-0],Pp ->■!],... ,P[Z ->&]), 

independently from the other groups. Therefore the vector 
in (4) is a sum of k + 1 independent multinomials. Now it 
is easy to compute both expectation and covariance. 

For a multinomial distribution (X 0 , X\ , . . . , Xk) with pro- 
babilities (po,Pi, • • • ,Pk), where X 0 + X\ + . . . + Xk = n, 
we have E X % = n ■ p % and 

Cov (X,, Xj) = E (X, <*pi)(Xj & Pj ) = n • (piS i=3 

In our case, X % = I's part of N ■ s'i, n = N ■ si, and 
Pi = p[l —t i]. For a sum of independent multinomial distri- 



butions, their expectations and covariances add together: 

E(N ■ s[,) = J2 N ■ Sl -P[l ^ I'] , 
Cov (N ■ s'i, N • s'j) = 

= Y,N. Sl .(p[l^i]. Si =J &p[l^i]. P [l^ j]) 



Thus, after dividing by an appropriate power of N, the for- 
mulae in the statement are proven. □ 



A.2 Proof of Statement 2 



PROOF. We are given a transaction t £ T and an itemset 
ACT, such that |t| = m, \A\ = k, and #(t f] A) = I. In 
the beginning of randomization, a number j is selected with 
distribution {p m [j]}, and this is what the first summation 
takes care of. Now assume that we retain exactly j items 
of t, and discard m items. 

Suppose there are q items from tCiA among the retained 
items. How likely is this? Well, there are (™) possible 
ways to choose j items from transaction t; and there are 
possible ways to choose q items from t n A and 
j <^q items from t \ A. Since all choices are equiproba- 
ble, we get Q / (™) as the probability that exactly q 
j4-items are retained. 

To make t' contain exactly I' items from A, we have 
to get additional V -&q items from A \ t. We know that 
#(A\t) = k<=d, and that any such item has probability p to 
get into t'. The last terms in (8) immediately follow. Sum- 
mation bounds restrict q to its actually possible (= nonzero 
probability) values. □ 



A.3 Proof of Statement 3 



PROOF. Let us denote 

pi :=(P[1^0],P[1^1],... ,P[l^k]) T , 

$ :=(g[Z<-0],g[Z<-l],... ,q[l^k]f. 

Since PQ = QP = I (where / is the identity matrix), we 



Notice also, from (7), that matrix D[l] can be written as 

D[l] = dmg(p l )^p l p l T , 
where diag(pi) denotes the diagonal matrix with pi-coord- 



inates as its diagonal elements. Now it is easy to see that 
s = q k T s' = '£ q [k^l'].s' l r, 
V*r~s=jjits l q k T D[l]q k = 

= E Sl $ k T ( dia s(Pi) &pipi T ) & = 

= Jj E Sl («* T dia s(Pi) 9* ^(Pi T &f) = 

= ^ E^E'Mn « [*<-«r 

(Var 5).. t = 

= jf E (* T ( E p P n 9 [ fc <- 2 = 
*E*-*«p«-*i) = £ E-KE ♦■'T* 

«*[*<-j]) = ^E *i(9[*«-i] a J])- 

□ 

A.4 Proof of Statement 4 

PROOF. We prove the left formula in (13) first, and then 
show that the right one follows from the left one. Consider 
N • Ei; it equals 

iV-E, = N-^2 supp T (C) = £ # {ti e T | C C ti} = 

CCA, |C| = ! CCA,|C| = ! 

= g#{cc^| |C| = i,ccti}. 

In other words, each transaction t t should be counted as 
many times as many different Z-sized subsets C C A it 
contains. From simple combinatorics we know that if j = 
#(A fl ti) and j ^ Z, then t x contains (^) different Z-sized 
subsets of A. Therefore, 

=e (;^#{t.eTi#(Ant,)=)}=E (;]"»^ 

and the left formula is proven. Now we can check the right 
formula just by replacing the 's according to the left for- 



mula. We have: 

g («)'-' S, = g («!)'"' (f) E (') = 

^r'iO(')-S'«s^-'(OC 5 ) 

= V" , 9 ! /^i v' = 

/L, *' Z!(q<^Z)! ^ 

since the sum ^ (<»1) J ' is zero whenever g^Z > 0. 

To prove that matrix P becomes lower triangular after 
the transformation from s and i" to E and E', let us find 
how E E' depends on E using the definition (12). 

E S;, = Esupp T '(C) = 

CCA, \C\ = [' 

= ^E E p "Ni']'^ = 
= E E^MnEMr'f;)w,T) = 

CCA, |C| = !' 1 = 0 3 = 1 \ / 

= EE(<*r'({Wp^ E ^(c,r) = 

J = 0 1 = 0 V / CCA, \C\ = V 

= E>; E E -pp t (s) = 

J = 0 CCA, |C| = I' SCC, \B\ = j 

= E C ^ E #{C|scccAiq = 0-su PP T (s) = 

J = 0 BCA, \B\=j 

Now it is clear that only the lower triangle of the matrix can 
have non-zeros. □ 
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International Business Machines Corporation Marc D. McSwain 

Research Division, Almaden Research Center 650 Harry Road, C4TA/J2B 

Intellectual Property Law Department San Jose, CA 95120-6099 

(408) 927-3364 
(408) 927-3375 - Fax 
mmcswain@us.ibm.com 



May 15, 2003 

Frederick W. Gibb m, Esq. 
McGinn & Gibb, PLLC 

Re: new docket ARC920030034US1 

Hi Fred: 

Please prepare and file a new patent application for this invention; a disclosure and a journal 
article describing the invention are attached. The article is available online at: 
http://www.almaden.ibm.com/cs/people/srikant/papers/kdd02.pdf 
and is generally cited as: 

A. Evfimievski, R. Srikant, R. Agrawal and J. Gehrke, "Privacy Preserving Mining of 
Association Rules," Proc. of 8th ACM SIGKDD Intl. Conf. on Knowledge Discovery and Data 
Mining (KDD), July 2002. 

The primary contact inventor is Ramakrishnan Srikant, who is at (408)927-1774, (408)927-3215 
fax, and srikant@almaden.ibm.com. The other inventors are Rakesh Agrawal and Alexandre 
Efvimiefski, who was a summer student IBM employee here when the invention was conceived. 
Professor Gehrke is listed as an author on the article, but is not an inventor. 

This invention was disclosed on July 23, 2002, so there's a statutory bar date approaching. 
We'll send you the references cited in the journal article and full inventor information separately. 
This invention is quite mathematical, so I would suggest placing the proofs into an Appendix. 
Dr. Srikant can help with reformatting the journal article so you can re-use as much of his 
material as directly as possible. 

Thanks, 
Marc 
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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re Application of 

Agrawal et al. Atty. Docket No. : ARC920030034US 1 

Serial No.: 10/624,069 Group Art Unit: 2161 

Filed: July 21, 2003 Examiner: Padmanabhan, Kavita 

For: MINING ASSOCIATION RULES OVER PRIVACY PRESERVING DATA 



PETITION UNDER 37 C.F.R. §1.147(a) TO PERMIT ACCEPTANCE OF 37 
C.F.R. AFFIDAVIT WITHOUT THE SIGNATURE OF ONE OF THE 
INVENTORS 



Director of the USPTO 
P.O. Box 1450 
Alexandria, VA 22313-1450 
Sir: 

Applicants hereby petition the Director to accept the 37 C.F.R. §1.131 declaration 
submitted as ATTACHMENT A with the simultaneously filed submission in support of 
the request for continued examination under 37 C.F.R. §1.114. The 37 C.F.R. §1.131 
declaration has been signed by two of the joint inventors of the present invention, 
namely, Alexandre Evfimievski and Ramakrishnan Srikant. Another of the joint 
inventors (Rakesh Agrawal) was at the time of the invention and at the time the present 
patent application was filed, an employee of the assignee, International Business 
Machines, Inc. However, Rakesh Agrawal no longer works for International Business 
Machines, Inc. and has refused to sign a 37 C.F.R. §1.131 declaration, which is necessary 
to overcome a prior art rejection. 
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Per the requirements of 37 C.F.R. §1.147 (a), the Applicants also submit that the 
last known work and home addresses of the non-signing inventor, Rakesh Aggrawal, was 
as follows: 

Work : 

Microsoft Research Labs 
1065 La Avenida 
Mountain View, CA 94043 

Home : 

1290 Quail Creek Circle 

San Jose, CA 95120Per the requirements of 37 C.F.R. §1.147 (a), the 

Applicants provide the attached statement (ATTACHMENT (a)) of Van Nguy, an 
attorney at the IBM Almaden Research Center in San Jose, California, along with copies 
of email correspondence between Van Nguy and Rakesh Agrawal (EXHIBITS (1)-(16) to 
Attachment (a)). The statement and emails show Van Nguy presented Rakesh Agrawal 
with three different versions of a 37 C.F.R. §1.131 declaration for his signature and that 
he refused to sign all three of the different versions. 

Also provided in support of Van Nguy's statement are copies of the three 
different versions of the 37 C.F.R. §1.131 declaration presented to Rakesh Agrawal by 
Van Nguy for his signature (see Attachments (b)-(d)). Attachment (b) differed from the 
executed declaration of Alexandre Evfimievski and Ramakrishnan Srikant only slightly. 
For example, Attachment (b) used the phrase "earliest effective prior art date", rather 
than the actual August 2002 date of Rizvi. Additionally, Attachment (b) contained the 
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statement "During all time periods mentioned herein and, specifically, between the 
conception date and the filing date of the Patent Application, all activities described 
herein occurred in the United States." 

Attachments (c) and (d) differed from the executed declaration of Alexandre 
Evfimievski and Ramakrishnan Srikant in that they were significantly shorter per Rakesh 
Agrawal's request. For example, Attachments (c) and (d) did not contain the listing of 
independent claims with reference to the exemplary locations within "Privacy Preserving 
Mining of Association Rules" (July 2002) (i.e., the published article of the inventors upon 
which the present application was based), wherein the claimed features are described. 
This listing was provided in the original version of the declaration presented to Rakesh 
Agrawal (i.e., Attachment (a)) and was retained in the executed declaration of Alexandre 
Evfimievski and Ramakrishnan Srikant. 

The Applicants submit that the statement of Van Nguy, as well as the other 
evidence provided, clearly indicate that Rakesh Agrawal is unwilling to or unable to 
continue to join in the prosecution of the present application. His refusal to sign a 37 
C.F.R. §1.131 affidavit was not based on a difference of opinion as to the facts of the 
case, but rather on Rakesh Agrawal's misunderstanding of the requirements of a 37 
C.F.R. §1.131 affidavit and, more specifically, his belief, despite efforts to persuade him 
otherwise, that a statement claiming that "our invention was done before July 2002" 
would be sufficient. Therefore, the Applicants respectfully request that the executed 
declaration of Alexandre Evfimievski and Ramakrishnan Srikant without the signature of 
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Rakesh Agrawal be accepted for purposes of establishing prior invention under 37 C.F.R. 

§1.131, as failure to do so will result in irreparable damage to the remaining inventors. 

Please charge the petition fee under § 1.17(g) of $200.00 for a §1.147 petition and 

any other fees required to Attorney's Deposit Account No. 09-0441. 

Respectfully submitted, 

Dated: 7/3/08 /Pamela M. Riley/ 

Pamela M. Riley 
Registration No. 40,146 

Gibb & Rahman, LLC 
2568-A Riva Road, Suite 304 
Annapolis, MD 21401 
Voice: (410) 573-0227 
Fax: (301) 261-8825 
Customer Number: 29154 
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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



attachment 



In re Application of 



Agrawal et al. 



Atty. Docket No.: ARC920030034US1 



Serial No.: 10/624,069 



Group Art Unit: 2161 



Filed: July 21, 2003 



Examiner: Padmanabhan, Kavita 



For: MINING ASSOCIATION RULES OVER PRIVACY PRESERVING DATA 

Commissioner of Patents 
P.O. BOX 1450 
Alexandria, VA 223 1 3- 1 450 

DECLARATION IN SUPPORT OF 37 C.F.R. 1.147(a) PETITION 

I, Van Nguy, an attorney with IBM, located at IBM Almaden Research Center, 
650 Harry Road, C4TA/J2B, San Jose, CA 95120, hereby declare the following: 

International Business Machines Corporation (IBM) is the assignee of the above- 
referenced patent application (herein after referred to as the Patent Application). Rakesh 
Agrawal is one of the joint inventors on the Patent Application. At the time of the 
invention that is the subject matter of the Patent Application and further at the time the 
Patent Application was filed, Rakesh Agrawal was an employee of IBM. Rakesh 
Agrawal is no longer employed by IBM. 

On May 28, 2008, I personally presented Rakesh Agrawal, via email, with an 
original version of a 37 C.F.R. §1.131 declaration, regarding the above-referenced patent 
application, for his signature. He refused to sign this original version. Specifically, over 
the course of several emails, he indicated that he was only willing to sign a "declaration 
that our invention was done before July 2002" and he questioned the use of terms of art 



such as "earliest effective prior art date." I tried to explain to him the purpose of the 
declaration, that all inventors are required to sign the declaration and that the limited 
statement he proposed may be insufficient. 

On June 3, 2008, after Rakesh Agrawal refused to sign the original version of the 
declaration, I presented him with second shorter of the declaration. Specifically, in this 
second version the listing of the independent claims along with the subsections of the 
July 2002 paper of the Applicants which disclosed the claimed features was eliminated. 
Rakesh Agrawal had questions regarding the language used in this second version and, 
therefore, on that same day I presented him with a third version of the declaration. 
Specifically, in this third version the phrase "earliest effective prior art date" was 
replaced with August 2002. Rakesh Agrawal continued to refuse to sign the declaration 
and indicated that he had "spent way to much time iterating" with me on it. 

The following is a timeline and a summary of the email correspondence I have 
had with Rakesh Agrawal regarding the 37 C.F.R. §1.131 declaration: 

May 15, 2008-1 emailed Rakesh Agrawal, asking if he would be "open to 
receiving and signing the declaration" (Exhibit 1 ). 

May 22, 2008-As I had received no response from Rakesh Agrawal, I again 
emailed him asking if he would be "open to receiving and signing the declaration" 
(Exhibit 2). 

May 22, 2008 — Rakesh Agrawal responded, asking the meaning of "you and your 
co-inventors invented before Rizvi" (Exhibit 3). 

May 23, 2008 — I responded to Rakesh Agrawal, answering his question and again 
asking if he was "open to receiving the declaration for signature" (Exhibit 4). 



May 24, 2008 — Rakesh Agrawal responded, asking "Isn't it obvious that our 
invention was done before August 2002 since the SIGKDD paper containing the 
invention was published in July 2002?" (Exhibit 5). 

May 27, 2008 — I responded to Rakesh Agrawal, answering his question and 
informing him that I would send him the declaration for his review and signature (Exhibit 
6). 

May 27, 2008 — Rakesh Agrawal responded, stating "I can sign the declaration 
that our was done before July 2002" (Exhibit 7). 

May 28, 2008 — I forwarded the original version of the declaration to Rakesh 
Agrawal for his signature and review (Exhibit 8). 

May 28, 2008 — Rakesh Agrawal responded, indicating "I thought I was simply 
declaring that: Ok, I can sign the declaration that our invention was done before July 
2002" (Exhibit 9). 

May 28, 2008 — I responded to Rakesh Agrawal, explaining the contents of the 
declaration (Exhibit 10). 

May 28, 2008 — Rakesh Agrawal responded, reiterating that he would only sign 
the declaration "that our invention was done before July 2002" (Exhibit 11). 

May 28, 2008 — I responded to Rakesh Agrawal, again explaining the contents of 
the declaration and informing him that the statement he proposed may be insufficient 
(Exhibit 12). 

June 3, 2008 — I forwarded a second shorter version of the declaration to Rakesh 
Agrawal (Exhibit 13), as discussed above. 



June 3, 2008 — Rakesh Agrawal responded to receipt of the second version, 
questioning the use of terms of art such as "earliest effective prior art date" and again 
indicated that he would not be willing to sign anything beyond "when we did the 
invention" (Exhibit 14). 

June 3, 2008 — I responded to Rakesh Agrawal, sending him a third version of the 
declaration that replaced the phrase "earliest effective prior art date" with August 2002 
(Exhibit 15), as discussed above. 

June 3, 2008 — Rakesh Agrawal responded stating, "I'm sorry I cannot sign this 
version. And frankly, I have spent way too much time iterating with you on it." (Exhibit 
16). 

I did not submit any additional versions of the declaration to Rakesh Agrawal for 
his signature, as he indicated that he was not receptive to receiving any version of the 
declaration from me that went beyond the statement "our invention was done before July 
2002." Furthermore, since 37 C.F.R. §1.131 requires a declaration that shows either 
reduction to practice prior to the effective date of the reference or conception of the 
invention prior to the effective date of the reference coupled with due diligence from 
prior to that date to the filing of the application, I believed that a declaration solely stating 
"our invention was done before July 2002" would be insufficient to remove the Rizvi 
article as a prior art reference. I believe Mr. Agrawal' s refusal to sign is based upon not 
understanding the legal reasons for submitting a declaration (e.g., see Exhibit 5), upon 
not understanding legal terms of art within the declaration (e.g., see Exhibit 14), and upon 
not understanding the factors that are weighed in determining whether a declaration is 
sufficient to remove a reference as prior art. I am not in a position to explain details of 



these matters to Mr. Agrawal as he no longer works for IBM and as I do not represent 
him as an individual. 

The above statements are made according to the best of my recollection, upon 
review of the appropriate documents and notes. I hereby declare that all statements made 
herein of my own knowledge are true and that all statements made on information and 
belief are believed to be true. I further hereby acknowledge that these statements were 
made with the knowledge that willful false statements and the like so made are 
punishable by fine or imprisonment, or both (18 U.S.C. 1001) and that such willful false 
statements may jeopardize the validity of the Patent Application or any patent issued 
thereon. 



Van NgiyT (Date) 



1-1 



Van Nguy/Almaden/IBM 
rakcsh.devnull(«^mB 
rakesh.agrawal@MBHM 

05/15/2008 03:09 PM 




rakesh.prof(2 




ragrawal@^HBBk 



DaUr 



Subject: *IBM Confidential: Signature needed to declare that you invented first 
Hi Rakesh - 

Re: US patent application 10/624,069 (Publication No 20050021488A1) 
Entitled: Mining Association Rules over Privacy Preserving Data 

The above case is still under examination at the United Stated Patent and Trademark Office (USPTO). The USPTO 
has cited an article (Rizvi et al) that was published before the filing of the above application, but after your article at 
ACM SIGKDD 2002 describing your invention. Rizvi actually cites your article as a reference. 

To confirm to the Patent Office that you and your co-inventors invented before Rizvi, we must file a declaration 
signed by all inventors declaring that you invented first. We would like to prepare the declaration for your signature. 
We wanted to make contact with you first to make sure that you are open to receiving and signing the declaration. 

Please let us know by THURSDAY MAY 22. 2008 if you approve us sending vou the declaration for signature. 

Best regards, 

Van N. Nguy 

Attorney. IBM Almaden Research Center 

650 Harry Road, C4TA/J2B, San Jose, CA 95120 



PREPARED BY OR FOR IBM ATTORNEY/PRIVILEGE REVIEW REQUIRED: This e-mail may contain 
privileged information and/or attorney work product, as it was prepared by or for an IBM attorney. Use or disclosure 
of such information by anyone other than a designated addressee is unauthorized. Do not copy or forward without 
authorization. If you are not an intended recipient, please notify the sender and delete this e-mail. 



EXHIBIT 1 



From: Van Nguy/Almaden/IBM 

ragrawal@MBQft rakesh.agrawal 



Date: 05/22/2008 03:42 PM 
Subject: Re: Signature needed to 



MBA rakesh.devnull@« 
that you invented first 



Hi Rakesh - 

I wanted to follow up on the email below since I have not heard from you. Are you open to us sending 
you a declaration to sign regarding the below? 

Best regards, 

Van N. Nguy 

Attorney, IBM Almaden Research Center 

650 Harry Road, C4TA/J2B, San Jose, CA 95120 



IBM CONFIDENTIAL 

PREPARED BY OR FOR IBM ATTORNEY/PRIVILEGE REVIEW REQUIRED: This e-mail may contain 
privileged information and/or attorney work product, as it was prepared by or for an IBM attorney. Use or 
disclosure of such information by anyone other than a designated addressee is unauthorized. Do not 
copy or forward without authorization. If you are not an intended recipient, please notify the sender and 
delete this e-mail. 



From: Van Nguy/Almaden/IBM 

To: rakesh.devnull@fgriMfe, rakesh.prof <§■■■■■■* ragrawal@BB| 

Date: 05/15/2008 03:09 PM 

Subject: "IBM Confidential: Signature needed to declare that you invented first 



Hi Rakesh - 

Re: US patent application 1 0/624,069 (Publication No 20050021 488A1 ) 
Entitled: Mining Association Rules over Privacy Preserving Data 

The above case is still under examination at the United Stated Patent and Trademark Office (USPTO). 
The USPTO has cited an article (Rizvi et al) that was published before the filing of the above application, 
but after your article at ACM SIGKDD 2002 describing your invention. Rizvi actually cites your article as a 
reference. 

To confirm to the Patent Office that you and your co-inventors invented before Rizvi, we must file a 
declaration signed by all inventors declaring that you invented first. We would like to prepare the 
declaration for your signature. We wanted to make contact with you first to make sure that you are open 
to receiving and signing the declaration. 

Please let us know by THURSDAY MAY 22. 2008 if vou approve us sending vou the declaration for 
signature. 

Best regards, 

Van N. Nguy 

Attorney, IBM Almaden Research Center 

650 Harry Road, C4TA/J2B, San Jose, CA 95120 



EXHIBIT 2 



PREPARED BY OR FOR IBM ATTORNEY/PRIVILEGE REVIEW REQUIRED: This e-mail may contain * 
privileged information and/or attorney work product, as it was prepared by or for an IBM attorney. Use or 
disclosure of such information by anyone other than a designated addressee is unauthorized. Do not 
copy or forward without authorization. If you are not an intended recipient, please notify the sender and 
delete this e-mail. 



EXHIBIT 3 



From: Rakesh Agrawal <Rakesh.Agrawal@ 

To: Van Ng uy/Almaden/IBM@IBMUS 

Date: 05/22/2008 1 1:10 PM 

Subject: RE: Signature needed to declare that you invented first 

you and your co-inventors invented before Rizvi — what does it mean? 



EXHIBIT 4 



From: Van Nguy HMMHHl 
Sent: Friday, May 23, 2008 9:31 AM 
To: Rakesh Agrawal 

Subject: RE: Signature needed to declare that you invented first 

Hi Rakesh - 

Thank you for your response. 

It means that the Patent Office thinks that the attached article by Rizvi et al. ("the Rizvi article") is "prior art" to your 
invention because the Rizvi article was published in August 2002 and your patent application (entitled "Mining 
Association Rules Over Privacy Preserving Data") was filed in July 2003. However, in the US, if we can show that 
you and your co-inventors invented before the effective date of the reference (here, August 20, 2002), then the 
reference is not really "prior art." 

One of the evidence we have that the Rizvi article is not prior art to your invention is a paper entitled "Privacy 
Preserving Mining of Association Rules" by you, your co-inventors (Alexandre Evfimievski and Ramakrishan 
Srikant) and Johannes Gehrke that includes a description of your the invention (as confirmed by Alexandre 
Evfimievski) and was published at ACM SIGKDD July 2002. The Rizvi article actually cites your article as 
reference [8]. 

The Patent Office requires, however, that to submit this evidence and to establish invention prior to the effective 
date of the reference (thus removing the Rizvi article as a "prior art"), we must submit a declaration signed by all the 
inventors saying as such. 

So, if you are open to receiving the declaration for signature, we will go ahead and prepare the document and send to 
you (which we can do over email) for your signature. Please let us know if you are open to this. 



(See attached file: p682-rizvi.pdf) (See attached file: p21 7-evfimievski.pdf) 
Best regards, 
Van N. Nguy 

Attorney, IBM Almaden Research Center 

650 Harry Road, C4TA/J2B, San Jose, CA 95 120 



IBM CONFIDENTIAL 

PREPARED BY OR FOR IBM ATTORNEY/PRIVILEGE REVIEW REQUIRED: This e-mail may contain 
privileged information and/or attorney work product, as it was prepared by or for an IBM attorney. Use or disclosure 
of such information by anyone other than a designated addressee is unauthorized. Do not copy or forward without 
authorization. If you are not an intended recipient, please notify the sender and delete this e-mail. 



EXHIBIT 5 



Rakesh Agrawal 
<Rakesh. Agrawal @4 

05/24/2008 02:22 PM ToVan Nguy/Almaden/IBM@IBMUS 



SubjectRE: Signature needed to declare that you invented first 



Isn't it obvious that our invention was done before August 2002 since the SIGKDD paper containing the 
invention was published in July 2002? 



EXHIBIT 6 



From: Van Nguy/Almaden/IBM 

To: ■■■■■■■■■■■■■■V, Rakesh Agrawal <Rakesh.Agrawal<§ 

Date: 05/27/2008 09:12 AM 

Subject: RE: Signature needed to declare that you invented first 



Hi Rakesh - 

Notwithstanding that, we cannot submit the evidence to the USPTO unless we have a declaration signed 
by all the inventors. Based on the emails below, I will assume that you are willing to receive the 
declaration for review and signature. I will go ahead and have it prepared. I will forward it onto you as 
soon as it is ready. 

Best regards, 

Van N. Nguy 

Attorney, IBM Almaden Research Center 

650 Harry Road, C4TA/J2B, San Jose, CA 95120 



IBM CONFIDENTIAL 

PREPARED BY OR FOR IBM ATTORNEY/PRIVILEGE REVIEW REQUIRED: This e-mail may contain 
privileged information and/or attorney work product, as it was prepared by or for an IBM attorney. Use or 
disclosure of such information by anyone other than a designated addressee is unauthorized. Do not 
copy or forward without authorization. If you are not an intended recipient, please notify the sender and 
delete this e-mail. 



EXHIBIT 7 



From: Rakesh Agraw al <Rakesh.Agrawal@flg(jm^ 

Cc: Van Nguy/Almaden/IBM@IBMUS 

Date: 05/27/2008 10:26 AM 

Subject: RE: Signature needed to declare that you invented first 

Ok, I can sign the declaration that our was done before July 2002. 



EXHIBIT 8 



From: Van Nguy [HMMMV] 
Sent: Wednesday, May 28, 2008 8:28 AM 
To: Rakesh Agrawal 

Subject: Fw: Signature needed to declare that you invented first 

Rakesh - Please find the attached for your review. Please sign the declaration and postal mail, email, or fax the 
original back to me at the number below. Best regards, Van 

(See attached file: Rizvi article.pd.fi (See attached file: ALM 5074 1.131 decfaration.doc)(See attached file: 
ARC920030034USl_Attachment C.pdfifSee attached file: privacy preserx'ing mining of association rules.pdf) 

Van N. Nguy 

Attorney, IBM Almaden Research Center 

650 Harry Road, C4TA/J2B, San Jose, CA 95120 



IBM CONFIDENTIAL 

PREPARED BY OR FOR IBM ATTORNEY/PRIVILEGE REVIEW REQUIRED: This e-mail may contain 
privileged information and/or attorney work product, as it was prepared by or for an IBM attorney. Use or disclosure 
of such information by anyone other than a designated addressee is unauthorized. Do not copy or forward without 
authorization. If you are not an intended recipient, please notify the sender and delete this e-mail. 



EXHIBIT 9 



Date: 
Subject: 



Rakesh Agrawal <Rakesh.Agrawal@i 
Van Nguy/Almader^IBI^ |BMUS^ 



05/28/2008 08:42 AM 
RE: Signature needed to declare that you invented first 



I thought I was simply declaring that: 

Ok, I can sign the declaration that our invention was done before July 2002. 



EXHIBIT 10 



From: Van Nguy ■■■■■i] 
Sent: Wednesday, May 28, 2008 1:22 PM 
To: Rakesh Agrawal 

Subject: RE: Signature needed to declare that you invented first ; 

Hi Rakesh - 

The declaration provides a detailed description of the evidence as they directly relate to the current claims since your 
"invention" is legally defined by the claims, though described, supported, and enabled by the specification. For your 
convenience, I have attached the declaration again here for your signature and return by postal mail, email, or fax to 
us. 

(See attached file: ALM 5074 1.131 declaration.doc) 

We look forward to hearing from you. 

Best regards, 
Van N. Nguy 

Attorney, IBM Almaden Research Center 

650 Harry Road, C4TA/J2B, San Jose, CA 95120 



IBM CONFIDENTIAL 

PREPARED BY OR FOR IBM ATTORNEY/PRIVILEGE REVIEW REQUIRED: This e-mail may contain 
privileged information and/or attorney work product, as it was prepared by or for an IBM attorney. Use or disclosure 
of such information by anyone other than a designated addressee is unauthorized. Do not copy or forward without 
authorization. If you are not an intended recipient, please notify the sender and delete this e-mail. 



EXHIBIT 11 



Rakesh Agrawal <Rakesh.Agrawal(S 
Van Nguy/Almaden/IBM@IBMUS 



Date: 05/28/2008 02:07 PM 
Subject: RE: Signature needed t< 



declare that you invented first 



Again: As I said earlier, I can sign the declaration that our invention was done before July 2002. The document 
you are asking me to sign seems to make representations beyond that!!! 



EXHIBIT 12 



From 
To: 



Van Nguy/Almaden/IBM 
Rakesh Agrawal <Rakesh.Agrawal( 




Date: 05/28/2008 04: 15 PM * 

Subject: RE: Signature needed to declare that you invented first 

Hi Rakesh - Yes. Unfortunately, the declaration needs to say what "our" and "invention" means in relation to the 
cited reference, so the declaration specifies this. The blanket statement you propose may be insufficient and we did 
not want to have to come back to you for another signature. We know you are very busy. Attached again is the 
declaration for your convenience. 

[attachment "ALM 5074 1.131 declaration.doc" deleted by Van Nguy/Almaden/IBM] 
Best regards, 
Van N. Nguy 

Attorney, IBM Almaden Research Center 

650 Harry Road, C4TA/J2B, San Jose, CA 95120 



IBM CONFIDENTIAL 

PREPARED BY OR FOR IBM ATTORNEY/PRIVILEGE REVIEW REQUIRED: This e-mail may contain 
privileged information and/or attorney work product, as it was prepared by or for an IBM attorney. Use or disclosure 
of such information by anyone other than a designated addressee is unauthorized. Do not copy or forward without 
authorization. If you are not an intended recipient, please notify the sender and delete this e-mail. 



EXHIBIT 13 



From: 

To: 

Cc: 



Rakesh Agrawal <Rakesh.Agrawal(^ 
Van Nguy/Almaden/IBM@IBMUS 





Dale; 



06/03/2008 12:38 PM 

RE: Signature needed to declare that you invented first 



What is meant by - earliest effective prior art date of Rizvi? 

I have no way of knowing when Rizvi et al conceived their innovation. There is usually a gap between when the 
invention is conceived and when a paper containing an invention is published. 

I told this to Laura - I can sign a declaration that says when we did the invention. Anything beyond, I am not 
comfortable. 

Also, please don't send anything marked IBM confidential to me. 



EXHIBIT 14 



From: Van Nguy [< 

Sent: Tuesday, June 03, 2008 1:54 PM 
To: Rakesh Agrawal 

cc: nmmKmammmmt^amam 

Subject: RE: Signature needed to declare that you invented first 



Hi Rakesh - 

I have asked the declaration be amended to remove references to terms of art (e.g., "earliest effective prior art date 
of Rizvi?") and replaced with an actual date (in this case, August 2002). So paragraph [0001] now begins: 

The purpose of this declaration is to prove that we conceived the claimed invention prior to the August 2002 date of 
Exhibit A. 

I hope this is a version meets with your approval. Thank you for all your attention to this matter. 
(See attached file: REVISED ALM 5074 1 131 declaration short version.doc) 
Best regards, 
Van N. Nguy 

Attorney, IBM Almaden Research Center 

650 Harry Road, C4TA/J2B, San Jose, CA 95 120 



PREPARED BY OR FOR IBM ATTORNEY/PRIVILEGE REVIEW REQUIRED: This e-mail may contain 
privileged information and/or attorney work product, as it was prepared by or for an IBM attorney. Use or disclosure 
of such information by anyone other than a designated addressee is unauthorized. Do not copy or forward without 
authorization. If you are not an intended recipient, please notify the sender and delete this e-mail. 



EXHIBIT 15 



From: Rakesh Agrawal <Rakesh.Agrawal@ 
To: Van N guy/Almaden/IBM@IBMUS 

Date: 06/03/2008 04:47 PM 

Subject: RE: Signature needed to declare that you invented first 

I'm sorry I cannot sign this version. And frankly, I have spent way too much time iterating with you on it. 



EXHIBIT 16 



attachment (b) 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



In re Application of 



Agrawal et al. 



Atty. Docket No.: ARC920030034US1 



Serial No.: 10/624,069 



Group Art Unit: 2161 



Filed: July 21, 2003 



Examiner: Padmanabhan, Kavita 



For: MINING ASSOCIATION RULES OVER PRIVACY PRESERVING DATA 



Commissioner of Patents 
P.O. BOX 1450 
Alexandria, VA 22313-1450 



We, Alexandre Evfimievski, Ramakrishnan Srikant, and Rakesh Agrawal, the Applicants 
and joint inventors of the above-referenced invention defined by claims 1-24 and disclosed in 
U.S. Patent Application Serial No. 10/624,069 hereby declare the following: 

[0001] The purpose of this declaration is to prove that we conceived the claimed 
invention prior to the earliest effective prior art date of Exhibit A. Exhibit A is a copy of the 
following published article cited in the March 5, 2008 rejection of claims 1-24 of the present 
patent application (herein after referred to as Patent Application) under 35 U.S.C. § 102(a): 
Rizvi, et al., "Maintaining Data Privacy in Association Rule Mining," Proceedings of the 28 th 
VLDB Conference, Hong Kong, China, 12 pages, dated August 2002 (hereinafter referred to as 
Rizvi). 



DECLARATION UNDER 37 C.F.R. §1.131 



[0002] The following shows that we conceived our invention prior to the August 2002 



earliest effective prior art date of Rizvi, that we were diligent from the date of conception to the 
date of reduction to practice and that we were further diligent to the date of the filing of the 
patent application (herein after referred to as Patent Application), which has a filing date of July 
21,2003. 

[0003] During all time periods mentioned herein and, specifically, between the 
conception date and the filing date of the Patent Application, all activities described herein 
occurred in the United States. 

[0004] Proof of the conception of the claimed invention prior to August 2002 and 
diligence in reducing the invention to practice and filing the Patent Application is demonstrated 
by the attached Exhibit B in conjunction with Exhibit A. 

[0005] Exhibit B is a copy of the following published paper: Evfimievski, R. Srikant, R. 
Agrawal and J. Gehrke, "Privacy Preserving Mining of Association Rules," Proc. Of 8 th ACM 
SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (KDD), July 2002, referred to 
herein as "Privacy Preserving Mining of Association Rules" (July 2002). 

[0006] Each of the Applicants of the Patent Application are co-authors on the paper 
"Privacy Preserving Mining of Association Rules" (July 2002) along with J. Gehrke. 

[0007] J. Gehrke was a professor and advisor of A. Evfimievski, during the time period 
in which the idea for the invention was conceived. Although J. Gehrke is listed as a co-author of 



2 



"Privacy Preserving Mining of Association Rules" (July 2002), he was not an inventor of the 
invention defined by claims 1-24 of the Patent Application. 

[0008] J. Gehrke has read the Patent Application and has declared that he is not an 
inventor of the invention defined by claims 1-24 (see Exhibit C). We, the Applicants, also 
acknowledge that J. Gehrke was not an inventor of the invention defined by claims 1-24 of the 
Patent Application. Therefore, the portions of "Privacy Preserving Mining of Association Rules" 
(July 2002), which describe the features of claims 1-24 of the Patent Application, describe the 
Applicants' own work and no one else's. 

[0009] "Privacy Preserving Mining of Association Rules" (July 2002) describes the 
invention defined by claims 1-24. In fact "Privacy Preserving Mining of Association Rules" 
(July 2002) served as the basis for the specification, drawings and claims of the Patent 
Application. 

[0010] The following is a listing of independent claims 1,7, 13, and 19 of the Patent 
Application that define the present invention with reference to the exemplary locations within 
"Privacy Preserving Mining of Association Rules" (July 2002), wherein the claimed feature is 
described: 

Claim 1 : A computer-implemented method of mining association rules over 
transactions from datasets while maintaining privacy of individual transactions within said 
datasets through randomization, said method comprising: 



3 



randomizing an original dataset to create a randomized dataset [see section 4.1], said 
randomizing comprising: 

randomly dropping true items from each transaction in said original dataset[see 
section 4.1]; and 

randomly inserting false items into each transaction in said original dataset[see 

section 4.1]; 

collecting said randomized dataset in a database [see section 4.1]; 

determining support of an association rule in said randomized dataset [see section 4.2]; 

estimating support of said association rule in said original dataset based on said support 
of said association rule in said randomized dataset [see section 4.3]; and 

outputting said association rule if said support of said association rule in said original 
data set is estimated to be greater than a predetermined minimum [see section 4.5], 

wherein, due to said randomizing, privacy breaches of said individual transactions are 
controlled [see section 4.4]. 

Claim 7 : A computer-implemented method of mining association rules from 
databases while maintaining privacy of individual transactions within said databases through 
randomization, said method comprising: 

randomizing an original dataset to create a randomized dataset [see section 4.1], said 
randomizing comprising: 

randomly dropping true items from each transaction in said original dataset [see 

section 4.1]; 



4 



randomly inserting false items into each transaction in said original dataset [see 

section 4.1]; 

collecting said randomized dataset in a database [see section 4.1]; 
mining said database to recover an association rule in said original dataset after said 
dropping and inserting processes, wherein said mining comprising [see sections 4.2-4.5]: 

determining support for said association rule in said randomized dataset [see 

section 4.2]; 

estimating support of said association rule in said original dataset based on said 
support of said association rule in said randomized dataset [see section 4.3]; and 

outputting said association rule if said support of said association rule in said 
original data set is estimated to be greater than a predetermined minimum [see section 4.5], 

wherein, due to said randomizing, privacy breaches of said individual transactions are 
controlled during said mining [see section 4.4]. 

Claim 13 : A computer-implemented method of mining association rules from 
datasets while maintaining privacy of individual transactions within said datasets through 
randomization, said method comprising: 

creating randomized transactions from an original dataset by [see section 4.1]: 

randomly dropping true items from each transaction in said original dataset [see 
section 4.1], and 

randomly inserting false items into each transaction in said original dataset [see 

section 4.1]; 



5 



creating a randomized dataset by collecting said randomized transactions [see section 

4.1]; 

collecting said randomized dataset in a database [see section 4.1]; and 
mining said database to recover an association rule in said original dataset after said 
dropping and inserting processes [see sections 4.2-4.5], wherein said mining comprises: 

determining support for said association rule ins aid randomized dataset [see 

section 4.2]; 

estimating support of said association rule in said original dataset based on said 
support for said association rule in said randomized dataset [see section 4.3]; and 

outputting said association rule if said support of said association rule in said 
original data set is estimated to be greater than a predetermined minimum [see section 4.5], 

wherein, due to said creating of said randomized transactions, privacy breaches of said 
individual transactions are controlled during said mining [see section 4.4]. 

Claim 19 : A computer program product on a computer-readable medium and 
tangibly embodying a program of instructions executable by a computer to perform a method of 
mining association rules from databases while maintaining privacy of individual transactions 
within said databases through randomization, said method comprising: 

randomizing an original dataset to create a randomized dataset [see section 4.1], said 
randomizing comprising: 

randomly dropping true items from each transaction in said original dataset [see 

section 4.1]; 



6 



randomly inserting false items into each transaction in said original dataset [see 

section 4.1]; 

collecting said randomized dataset in a database [see section 4.1]; and 
mining said database to recover an association rule in said original dataset after said 
dropping and inserting processes [see sections 4.2-4.5], wherein said mining comprises: 

determining support for said association rule in said randomized dataset [see 

section 4.2]; 

estimating support of said association rule in said original dataset based on said 
support of said association rule in said randomized dataset [see section 4.3]; and 

outputting said association rule if said support of said association rule in said 
original data set is estimated to be greater than a predetermined minimum [see section 4.5], 

wherein, due to said randomizing, privacy breaches of said individual transactions are 
controlled during said mining [see section 4.4]. 

[0011] Furthermore, dependent claims 2-6, 8-12, 14-18 and 20-24 are either explicitly 
described in "Privacy Preserving Mining of Association Rules" (July 2002) or inferred from 
details contained therein. 

[0012] "Privacy Preserving Mining of Association Rules" (July 2002) clearly predates 
the August 2002 earliest effective prior art date of Rizvi. Additionally, at the August 2002 
earliest effective prior art date of Rizvi, the authors of Rizvi had knowledge of the details of the 
present invention and wrote their paper in light of that knowledge. This is evidenced by the fact 
that, as mentioned above, the details of the invention as defined by claims 1-24 of the Patent 



7 



Application are described in "Privacy Preserving Mining of Association Rules" (July 2002) and 
further by the fact that Rizvi cites "Privacy Preserving Mining of Association Rules" (July 
2002), as a reference, at various places throughout the text of the article. 

[0013] We were diligent from the date of conception in reducing the invention to practice 
and in pursuing, preparing, and filing the Patent Application. 

[0014] On May 15, 2003, a patent attorney was instructed to prepare a patent application 
that eventually became the Patent Application. The Patent Application was eventually prepared 
and filed on July 21, 2003. 

[0015] Finally, the above declarations are made according to the best of my/our 
recollection upon review of the appropriate documents and notes, and I/we hereby acknowledge 
that willful false statements and the like are punishable by fine or imprisonment, or both (18 
U.S.C. 1001) and may jeopardize the validity of the Patent Application or any patent issuing 
thereon. All statements that are made herein of my/our own knowledge are true and all 
statements that are made herein based on information and belief are believed to be true. 



Alexandre Evfimievski 


(Date) 


Ramakrishnan Srikant 


(Date) 



Rakesh Agrawal (Date) 
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attachment (c) 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



In re Application of 



Agrawal et al. 



Atty. Docket No.: ARC920030034US1 



Serial No.: 10/624,069 



Group Art Unit: 2161 



Filed: July 21, 2003 



Examiner: Padmanabhan, Kavita 



For: MINING ASSOCIATION RULES OVER PRIVACY PRESERVING DATA 



Commissioner of Patents 
P.O. BOX 1450 
Alexandria, VA 22313-1450 



We, Alexandre Evfimievski, Ramakrishnan Srikant, and Rakesh Agrawal, the Applicants 
and joint inventors of the above-referenced invention defined by claims 1-24 and disclosed in 
U.S. Patent Application Serial No. 10/624,069 hereby declare the following: 

[0001] The purpose of this declaration is to prove that we conceived the claimed 
invention prior to the earliest effective prior art date of Exhibit A. Exhibit A is a copy of the 
following published article cited in the March 5, 2008 rejection of claims 1-24 of the present 
patent application (herein after referred to as Patent Application) under 35 U.S.C. § 102(a): 
Rizvi, et al., "Maintaining Data Privacy in Association Rule Mining," Proceedings of the 28 th 
VLDB Conference, Hong Kong, China, 12 pages, dated August 2002 (hereinafter referred to as 
Rizvi). 



DECLARATION UNDER 37 C.F.R. §1.131 



[0002] The following shows that we conceived our invention prior to the August 2002 



earliest effective prior art date of Rizvi, that we were diligent from the date of conception to the 
date of reduction to practice and that we were further diligent to the date of the filing of the 
patent application (herein after referred to as Patent Application), which has a filing date of July 
21,2003. 

[0003] Proof of the conception of the claimed invention prior to August 2002 and 
diligence in reducing the invention to practice and filing the Patent Application is demonstrated 
by the attached Exhibit B in conjunction with Exhibit A. 

[0004] Exhibit B is a copy of the following published paper: Evfimievski, R. Srikant, R. 
Agrawal and J. Gehrke, "Privacy Preserving Mining of Association Rules," Proc. Of 8 th ACM 
SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (KDD), July 2002, referred to 
herein as "Privacy Preserving Mining of Association Rules" (July 2002). 

[0005] Each of the Applicants of the Patent Application are co-authors on the paper 
"Privacy Preserving Mining of Association Rules" (July 2002) along with J. Gehrke. 

[0006] J. Gehrke was a professor and advisor of A. Evfimievski, during the time period 
in which the idea for the invention was conceived. Although J. Gehrke is listed as a co-author of 
"Privacy Preserving Mining of Association Rules" (July 2002), he was not an inventor of the 
invention defined by claims 1-24 of the Patent Application. 

[0007] J. Gehrke has read the Patent Application and has declared that he is not an 
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inventor of the invention defined by claims 1-24 (see Exhibit C). We, the Applicants, also 
acknowledge that J. Gehrke was not an inventor of the invention defined by claims 1-24 of the 
Patent Application. Therefore, the portions of "Privacy Preserving Mining of Association Rules" 
(July 2002), which describe the features of claims 1-24 of the Patent Application, describe the 
Applicants' own work and no one else's. 

[0008] "Privacy Preserving Mining of Association Rules" (July 2002) describes the 
invention defined by claims 1-24. In fact "Privacy Preserving Mining of Association Rules" 
(July 2002) and, in particular, section 4 of "Privacy Preserving Mining of Association Rules" 
(July 2002), served as the basis for the specification, drawings and claims of the Patent 
Application. 

[0009] Furthermore, dependent claims 2-6, 8-12, 14-18 and 20-24 are either explicitly 
described in "Privacy Preserving Mining of Association Rules" (July 2002) or inferred from 
details contained therein. 

[0010] "Privacy Preserving Mining of Association Rules" (July 2002) clearly predates 
the August 2002 earliest effective prior art date of Rizvi. Additionally, at the August 2002 
earliest effective prior art date of Rizvi, the authors of Rizvi had knowledge of the details of the 
present invention and wrote their paper in light of that knowledge. This is evidenced by the fact 
that, as mentioned above, the details of the invention as defined by claims 1-24 of the Patent 
Application are described in "Privacy Preserving Mining of Association Rules" (July 2002) and 
further by the fact that Rizvi cites "Privacy Preserving Mining of Association Rules" (July 
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2002), as a reference, at various places throughout the text of the article. 

[0011] We were diligent from the date of conception in reducing the invention to practice 
and in pursuing, preparing, and filing the Patent Application. 

[0012] On May 15, 2003, a patent attorney was instructed to prepare a patent application 
that eventually became the Patent Application. The Patent Application was eventually prepared 
and filed on July 21, 2003. 

[0013] Finally, the above declarations are made according to the best of my/our 
recollection upon review of the appropriate documents and notes, and I/we hereby acknowledge 
that willful false statements and the like are punishable by fine or imprisonment, or both (18 
U.S.C. 1001) and may jeopardize the validity of the Patent Application or any patent issuing 
thereon. All statements that are made herein of my/our own knowledge are true and all 
statements that are made herein based on information and belief are believed to be true. 



Alexandre Evfimievski 


(Date) 


Ramakrishnan Srikant 


(Date) 



Rakesh Agrawal (Date) 
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attachment (d) 

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



In re Application of 



Agrawal et al. 



Atty. Docket No.: ARC920030034US1 



Serial No.: 10/624,069 



Group Art Unit: 2161 



Filed: July 21, 2003 



Examiner: Padmanabhan, Kavita 



For: MINING ASSOCIATION RULES OVER PRIVACY PRESERVING DATA 



Commissioner of Patents 
P.O. BOX 1450 
Alexandria, VA 22313-1450 



We, Alexandre Evfimievski, Ramakrishnan Srikant, and Rakesh Agrawal, the Applicants 
and joint inventors of the above-referenced invention defined by claims 1-24 and disclosed in 
U.S. Patent Application Serial No. 10/624,069 hereby declare the following: 

[0001] The purpose of this declaration is to prove that we conceived the claimed 
invention prior to the August 2002 date of Exhibit A. Exhibit A is a copy of the following 
published article cited in the March 5, 2008 rejection of claims 1-24 of the present patent 
application (herein after referred to as Patent Application) under 35 U.S.C. § 102(a): Rizvi, et al., 
"Maintaining Data Privacy in Association Rule Mining," Proceedings of the 28 th VLDB 
Conference, Hong Kong, China, 12 pages, dated August 2002 (hereinafter referred to as Rizvi). 



DECLARATION UNDER 37 C.F.R. §1.131 



[0002] The following shows that we conceived our invention prior to the August 2002 



date of Rizvi, that we were diligent from the date of conception to the date of reduction to 
practice and that we were further diligent to the date of the filing of the patent application (herein 
after referred to as Patent Application), which has a filing date of July 21, 2003. 

[0003] Proof of the conception of the claimed invention prior to August 2002 and 
diligence in reducing the invention to practice and filing the Patent Application is demonstrated 
by the attached Exhibit B in conjunction with Exhibit A. 

[0004] Exhibit B is a copy of the following published paper: Evfimievski, R. Srikant, R. 
Agrawal and J. Gehrke, "Privacy Preserving Mining of Association Rules," Proc. Of 8 th ACM 
SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (KDD), July 2002, referred to 
herein as "Privacy Preserving Mining of Association Rules" (July 2002). 

[0005] Each of the Applicants of the Patent Application are co-authors on the paper 
"Privacy Preserving Mining of Association Rules" (July 2002) along with J. Gehrke. 

[0006] J. Gehrke was a professor and advisor of A. Evfimievski, during the time period 
in which the idea for the invention was conceived. Although J. Gehrke is listed as a co-author of 
"Privacy Preserving Mining of Association Rules" (July 2002), he was not an inventor of the 
invention defined by claims 1-24 of the Patent Application. 

[0007] J. Gehrke has read the Patent Application and has declared that he is not an 
inventor of the invention defined by claims 1-24 (see Exhibit C). We, the Applicants, also 
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acknowledge that J. Gehrke was not an inventor of the invention defined by claims 1-24 of the 
Patent Application. Therefore, the portions of "Privacy Preserving Mining of Association Rules" 
(July 2002), which describe the features of claims 1-24 of the Patent Application, describe the 
Applicants' own work and no one else's. 

[0008] "Privacy Preserving Mining of Association Rules" (July 2002) describes the 
invention defined by claims 1-24. In fact "Privacy Preserving Mining of Association Rules" 
(July 2002) and, in particular, section 4 of "Privacy Preserving Mining of Association Rules" 
(July 2002), served as the basis for the specification, drawings and claims of the Patent 
Application. 

[0009] "Privacy Preserving Mining of Association Rules" (July 2002) clearly predates 
the August 2002 date of Rizvi, which in fact cites "Privacy Preserving Mining of Association 
Rules" (July 2002), as a reference, at various places throughout the text of the article. 

[0010] We were diligent from the date of conception of our invention in reducing the 
invention to practice and in pursuing, preparing, and filing the Patent Application. 

[0011] On May 15, 2003, a patent attorney was instructed to prepare a patent application 
that eventually became the Patent Application. The Patent Application was eventually prepared 
and filed on July 21, 2003. 

[0012] Finally, the above declarations are made according to the best of my/our 
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recollection upon review of the appropriate documents and notes, and I/we hereby acknowledge 
that willful false statements and the like are punishable by fine or imprisonment, or both (18 
U.S.C. 1001) and may jeopardize the validity of the Patent Application or any patent issuing 
thereon. All statements that are made herein of my/our own knowledge are true and all 
statements that are made herein based on information and belief are believed to be true. 



Alexandre Evfimievski (Date) 



Ramakrishnan Srikant (Date) 



Rakesh Agrawal (Date) 
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